Wes, This makes sense. Thank you.
On Wed, Dec 28, 2016 at 10:19 AM, Wes McKinney <[email protected]> wrote: > hi Deepak, > > On Wed, Dec 28, 2016 at 9:24 AM, Deepak Majeti <[email protected]> > wrote: > > Wes, > > > > I too agree that having a single general purpose I/O implementation makes > > sense moving forward. > > We can have a separate build option to build only the parquet-cpp library > > for those who want to encode/decode just the parquet format. > > The changes I'm describing are more invasive because it also coupled > to parquet-cpp's memory management, namely parquet::Buffer > (https://github.com/apache/parquet-cpp/blob/master/src/ > parquet/util/buffer.h) > and parquet::MemoryAllocator > (https://github.com/apache/parquet-cpp/blob/master/src/ > parquet/util/mem-allocator.h). > > One of the benefits of consolidating the memory interfaces is that we > can provide zero-copy data handoffs from the decoders to Arrow > consumers (or other applications that handle the buffers). > > The goal is that we would make it easy for a parquet-cpp user to > statically link only the libarrow symbols she/he needs for the > application and not have to ship Arrow shared libraries separately > necessarily. > > Let me know if this makes sense / what concerns you might have. > > Thanks > Wes > > > > > > > On Wed, Dec 28, 2016 at 9:04 AM, Uwe L. Korn <[email protected]> wrote: > > > >> Hello Wes, > >> > >> As the classes used from Arrow are very low-level and should not > >> prohibit the integration of parquet-cpp into other systems, I'm fully in > >> support of making Arrow a hard dependency. The current two > >> implementations look very similar of the not-shared-but-shall-be-shared > >> components. Given that I/O implementations in Arrow would also benefit > >> (from) other file formats such as Feather and the memory management of > >> Arrow will see great improvements I don't see a need that parquet-cpp > >> should have its own abstractions. > >> > >> Cheers > >> Uwe > >> > >> -- > >> Uwe L. Korn > >> [email protected] > >> > >> On Fri, Dec 23, 2016, at 11:17 PM, Wes McKinney wrote: > >> > hi folks, > >> > > >> > Spurred by the discussion and bugfix for PARQUET-799, I'd like to do > >> > something about the IO interfaces that we currently have implemented > >> > in parquet-cpp. > >> > > >> > For C++ at least, the Parquet project is not an ideal place to be > >> > maintaining cross-platform IO and memory management. There are > >> > portability and concurrent access issues we will eventually need to > >> > deal with to make parquet-cpp work well in diverse production > >> > environments. > >> > > >> > In parallel, we've been developing a general, low-overhead IO > >> > subsystem inside Apache Arrow: > >> > > >> > https://github.com/apache/arrow/tree/master/cpp/src/arrow/io > >> > > >> > Since Arrow is about in-memory columnar data structures and efficient > >> > IO / RPC / IPC, this is a much more appropriate place to maintain such > >> > code (in the absence of a sort of "Apache C++ Commons" library). > >> > There, we currently have more mature implementations of: > >> > > >> > - Operating system files (which also work on Windows) > >> > - Memory mapped files > >> > - HDFS (either using libhdfs or libhdfs3 at your choosing) > >> > > >> > Additionally, the "Buffer" abstraction (which handles memory lifetime > >> > and provides a general-purpose way to pass around a block of memory > >> > which may or may not be owned by the application) is implemented in > >> > both Parquet [1] and Arrow [2]. > >> > > >> > Since, fundamentally, parquet-cpp is a library for encoding and > >> > decoding the Parquet file format rather than general purpose IO / > >> > file-like interfaces, I propose that we excise this code from the > >> > library and make Arrow a hard dependency in libparquet. I believe our > >> > respective developer communities would benefit from a hardening of the > >> > IO and memory interfaces that are being developed in Arrow, and it > >> > will lead to better quality software and reduced fragmentation. > >> > > >> > I wanted to bring this up as we are on the cusp of making the first > >> > ASF release of parquet-cpp, and while this work might not make the cut > >> > for 0.1, if we agree it's a good idea it would be good to do it sooner > >> > rather than later. > >> > > >> > Thanks and happy holidays / best wishes for 2017, > >> > Wes > >> > > >> > [1]: > >> > https://github.com/apache/parquet-cpp/blob/master/src/ > >> parquet/util/buffer.h > >> > [2]: https://github.com/apache/arrow/blob/master/cpp/src/ > arrow/buffer.h > >> > > > > > > > > -- > > regards, > > Deepak Majeti > -- regards, Deepak Majeti
