[
https://issues.apache.org/jira/browse/MNEMONIC-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192563#comment-15192563
]
Yanping Wang commented on MNEMONIC-2:
-------------------------------------
Move Arrow integration discussion here as part of record.
===============================
From: Jacques Nadeau [mailto:[email protected]]
Sent: Friday, March 11, 2016 11:41 AM
To: Wang, Gang1 <[email protected]>
Cc: Jacques Nadeau <[email protected]>; Wang, Yanping <[email protected]>
Subject: Re: [DISCUSS] Mnemonic - Arrow Integration
Yup, that should work.
On Thu, Mar 10, 2016 at 5:28 PM, Wang, Gang1 <[email protected]> wrote:
Hi Jacques,
Thank you very much for your guidance and your advices are very helpful for
me to understand the internals of Apache Arrow, we will follow up and try to
create a MnmonicByteBuf similar to the class
io.netty.buffer.UnpooledUnsafeDirectByteBuf for class LargeBuffer if I
understand correctly, Thanks.
Best Regards
Gary (Wang, Gang)
From: Jacques Nadeau [mailto:[email protected]]
Sent: Thursday, March 10, 2016 4:05 PM
To: Wang, Gang1 <[email protected]>
Cc: Jacques Nadeau <[email protected]>; Wang, Yanping <[email protected]>
Subject: Re: [DISCUSS] Mnemonic - Arrow Integration
We should actually link lower. ArrowBuf is a finalized single implementation
for performance reasons. We should stitch in at the constructor level here:
https://github.com/apache/arrow/blob/master/java/memory/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java
This allows us to manage accounting, sharing etc above the physical layer. Note
how this class has a number of different representations of data it can wrap
including PooledUnsafeDirectByteBuf, LargeBuffer and DuplicatedByteBuf.
The main things that we need to have inside the new implementation are: how to
release the memory, the starting address and the length. Read and write is done
directly with memory addresses.
On Thu, Mar 10, 2016 at 3:54 PM, Wang, Gang1 <[email protected]> wrote:
Hi Jacques,
After further code investigation, we found the interface BufferAllocator is a
better point to back Apache Arrow using Mnemonic infrastructure as an
alternatives.
However, the interface of BufferAllocator in turn depends on the class ArrowBuf
that is not an interface so, for the purpose of extension and separating of
concerns, is it possible to convert the ArrowBuf to an interface officially for
Apache Arrow ? surely, we can do it if desired, Thanks.
Best Regards
Gary (Wang, Gang)
> Integration Mnemonic and Apache Arrow, Phase 1
> ----------------------------------------------
>
> Key: MNEMONIC-2
> URL: https://issues.apache.org/jira/browse/MNEMONIC-2
> Project: Mnemonic
> Issue Type: Improvement
> Reporter: Wang, Gang
> Labels: features
>
> as we have discussed, Mnemonic is able to improve the generality of Apache
> Arrow, so I create this issue to address this new feature for discussion and
> task tracking, Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
