Re: MemoryPool in Arrow libraries

2019-03-21 Thread Uwe L. Korn
Hello, > On alignment: The Arrow Spec calls for at least 8-byte alignment but > recommends 64-byte alignment precisely for SIMD use-cases. There is still > an open JIRA item [3] to make Java have 64-byte alignment, so I don't think > Java is handling 64-byte alignment (I don't know about 8-byte

Re: MemoryPool in Arrow libraries

2019-03-20 Thread Micah Kornfield
Hi Eric, I'm not an expert on most of this, but hopefully this will add a little bit of context. My sense is that the Java implementation is farther along for "real world" use-cases because it is used by Dremio and Drill in distributed analytics engines, where they've had to solve find grained mem

MemoryPool in Arrow libraries

2019-03-18 Thread Eric Erhardt
We are having a discussion on https://github.com/apache/arrow/pull/3925#issuecomment-473605919 about the `MemoryPool` class in the C# library. In reality, the way `MemoryPool` is designed in C#, it is more of a "MemoryAllocator" - it just allocates or reallocates memory. There is no API for "r