Agree and I understand what you concerned about that, however we hope those underlying distributed object graph could be transparent for upper layer data structures or better with some hints :), but anyway, we could firstly focus on code moving and stabilization for Mnemonic as you suggested, Thanks.
On 3/24/2016 2:03 AM, Zheng, Kai wrote: > Hi Gary, > > Thanks for this work and the nice proposal > > Overall it makes sense. I do have a concern over this item: >>> - Arrow could take advantages of coming Mnemonic features of memory >>> clustering/DOG (distributed object graph) and massive native computing > I would think this will go too far away from the initiatives it should focus > on, at least in medium future. There are bundles of projects that provide > distributed data structures, and Mnemonic may be better to be utilized in > them as a memory centric library project. > > By the way, I thought we should try to boost this project up in the ASF space > and have the first release out as soon as possible. Is there any plan or > update for that? > > Note Apache Arrow itself is still in the very early stage. The integration > would be good to happen when the both have their first releases out. > > Regards, > Kai > > -----Original Message----- > From: Gary [mailto:[email protected]] > Sent: Thursday, March 24, 2016 10:19 AM > To: [email protected] > Subject: A Proposal Apache Incubator Mnemonic as an alternative infra. for > Apache Arrow > > Hello, > > We have created a patch for Apache Arrow to leverage Apache incubator > Mnemonic as an alternative infra. for underlying memory resources allocation, > you can find it as below forked repo. > > https://github.com/NonVolatileComputing/arrow > > By this way, Apache Arrow could take some structural benefits from > Mnemonic project they are > > - Arrow is able to leverage larger capacity of high performance hybrid > storage devices. e.g. high-end SSD, NVMe > > - Mnemonic provide a potential opportunity for Arrow to optimize/tuning > its allocation algorithms as a native Arrow-oriented allocation services > > - The non-volatile features of Mnemonic make it possible that Arrow > could make its columnar in-memory data shared between different applications > or across life-cycle of single application > > - Arrow could take advantages of coming Mnemonic features of memory > clustering/DOG (distributed object graph) and massive native computing > > - Mnemonic helps to reduce the pressure of main memory utilization and > its related system wide overheads. > > Our this patch is designed to minimize the changes for user to use Arrow, > please check out the test cases provided by this patch for your reference. > > Note that, we need to put allocator services to a specified position > (indicated by pom.xml) for Mnemonic backed Arrow related test cases to run > because those services are required for external memory-like device > management. > > Please give your comments and review feedback for better collaboration of > Apache Arrow and Mnemonic, Thanks. > > Best Regards. > Gary. > > > >
signature.asc
Description: OpenPGP digital signature
