Hi Gary, Thanks for this work and the nice proposal
Overall it makes sense. I do have a concern over this item: >> - Arrow could take advantages of coming Mnemonic features of memory >> clustering/DOG (distributed object graph) and massive native computing I would think this will go too far away from the initiatives it should focus on, at least in medium future. There are bundles of projects that provide distributed data structures, and Mnemonic may be better to be utilized in them as a memory centric library project. By the way, I thought we should try to boost this project up in the ASF space and have the first release out as soon as possible. Is there any plan or update for that? Note Apache Arrow itself is still in the very early stage. The integration would be good to happen when the both have their first releases out. Regards, Kai -----Original Message----- From: Gary [mailto:[email protected]] Sent: Thursday, March 24, 2016 10:19 AM To: [email protected] Subject: A Proposal Apache Incubator Mnemonic as an alternative infra. for Apache Arrow Hello, We have created a patch for Apache Arrow to leverage Apache incubator Mnemonic as an alternative infra. for underlying memory resources allocation, you can find it as below forked repo. https://github.com/NonVolatileComputing/arrow By this way, Apache Arrow could take some structural benefits from Mnemonic project they are - Arrow is able to leverage larger capacity of high performance hybrid storage devices. e.g. high-end SSD, NVMe - Mnemonic provide a potential opportunity for Arrow to optimize/tuning its allocation algorithms as a native Arrow-oriented allocation services - The non-volatile features of Mnemonic make it possible that Arrow could make its columnar in-memory data shared between different applications or across life-cycle of single application - Arrow could take advantages of coming Mnemonic features of memory clustering/DOG (distributed object graph) and massive native computing - Mnemonic helps to reduce the pressure of main memory utilization and its related system wide overheads. Our this patch is designed to minimize the changes for user to use Arrow, please check out the test cases provided by this patch for your reference. Note that, we need to put allocator services to a specified position (indicated by pom.xml) for Mnemonic backed Arrow related test cases to run because those services are required for external memory-like device management. Please give your comments and review feedback for better collaboration of Apache Arrow and Mnemonic, Thanks. Best Regards. Gary.
