> > I > wonder if there might be mileage in building off an existing format > instead of defining another flatbuffer based tabular encoding >
We *are* building off an existing format. Parquet will stay as it is, its logical structure and everything. The encoding used for this logical structure's metadata is just a detail. In contrast, Arrow (Feather) is missing so many things that Parquet has, so you would need a lot more work to get to where Parquet is, if you started off with Arrow. I also wonder if, for example, feather added better encoding > support, we would get something that met all the requirements, without > needing to effectively create a third file format? I would argue that feather is faaaaar farther away from being a good storage format than Parquet is. Yes, you could add a lot of stuff to feather to make it a great storage format, but it would be way more than you would need to add to Parquet to make it as good. And you would also be creating de facto a third format if you change that much. That's also why Feather has little adoption: It's not a good storage format as of now, as its based on Arrow and Arrow isn't design for that. One thing that occurs to me from reading a number of the proposals / > investigations in this space is how similar many of them end up looking > to Apache Arrow Feather / IPC. I would argue that Parquet and Arrow are actually not that similar. They are similar, but only "trivially so" (i.e., in ways in which each storage format would end up at a roughly similar design): Obvious similarities: * Most modern formats are chunked columnar, as this is just obviously a good idea for many scenarios, as you can choose to skip columns and chunking helps with being able to write without having to buffer all the file before writing it. In that regard Parquet and Arrow are similar, but so are ORC and others. * Formats have some kind of metadata structure describing the file. In that regard, Parquet and Arrow and all others are similar, but every format needs this, so the similarity is trivial. * Both have encodings. Again, any format needs this. Also the fact that there is overlap in the encodings is just because some encodings are an obviously good idea (e.g., dictionary). * Both have a footer. Any format that wants to be able to be written in a streaming fashion needs this, you don't know the metadata of the file yet when you start writing, so you cannot use a header and putting the metadata somewhere in the middle would make it hard to discover. Now, the metadata structure needs to be encoded, somehow. For this, modern formats use encoding standards like thrift or flatbuffers. But I would argue that the actual encoding of the metadata is *not* what defines a format, at all; it is just a means to encode the logical structure, and while this structure has some resemblance between Arrow and Parquet, there are actually a lot of differences. Each format somehow has to do it and just because two formats may decide to use the same encoding (e.g., flatbuffers) that doesn't make them very similar. But here is the greatest dissimilarity: Arrow was made to be an IPC streaming format. As such, it needs to be very fast to read and write. Then the python guys came and invented feather to make the whole thing also work somewhat as a storage format, but all they did was basically add a footer; the format itself (i.e., the encodings) are not at all optimized for a storage format. A storage format may / should use more heavyweight encodings and it is okay if it is more costly to write, as you expect it to be read multiple times while an IPC format is usually only read once. Could Arrow become more of a storage format by adding more heavyweight encodings, statistics, and indexes? It sure could! And maybe it will, but that's not what it was designed for, so I doubt that such things will be a high priority for Arrow. But I wouldn't - at least as of now - want to base a storage format on Arrow. There is just too much missing. A lot of things that Parquet has that Arrow hasn't (all the statistics, bloom filters, indexes, etc.). It may look like Parquet is becoming Arrow if it adopts flatbuffers as a metadata format, but I would argue that is not at all the case. The metadata encoding is just a tiny detail and a lot of other things are what makes Parquet Parquet. Cheers, Jan Am Do., 15. Aug. 2024 um 15:20 Uhr schrieb Raphael Taylor-Davies <r.taylordav...@googlemail.com.invalid>: > Hi, > > One thing that occurs to me from reading a number of the proposals / > investigations in this space is how similar many of them end up looking > to Apache Arrow Feather / IPC. Parquet does have a narrower type system > and broader encoding and statistics support, but as far as the > underlying file structure is concerned feather looks a lot like many of > the parquet v3 proposals. > > Despite this feather has not seen especially broad adoption as far as I > am aware, and I therefore wonder if there might be relevant learnings > here? I also wonder if, for example, feather added better encoding > support, we would get something that met all the requirements, without > needing to effectively create a third file format? > > To be clear I have no particular affection for feather, in fact I find > the way it handles dictionaries to be especially distasteful, but I > wonder if there might be mileage in building off an existing format > instead of defining another flatbuffer based tabular encoding... > > Kind Regards, > > Raphael > > On 15/08/2024 13:41, Jan Finis wrote: > > I guess most close source implementations have done these optimizations > > already, it has just not been done in the open source versions. E.g., we > > switched to a custom-built thrift runtime using pool allocators and > string > > views instead of copied strings a few years ago, seeing comparable > > speed-ups. The C++ thrift library is just horribly inefficient. > > > > I agree with Alkis though that there are some gains that can be achieved > by > > optimizing, but the format has inherent drawbacks. Flatbuffers is indeed > > more efficient but at the cost of increased size. > > Alkis, can you elaborate how you brought the size of Flatbuffers down? > > > > Cheers, > > Jan > > > > Am Do., 15. Aug. 2024 um 13:50 Uhr schrieb Andrew Lamb < > > andrewlam...@gmail.com>: > > > >> I don't disagree that flatbuffers would be faster than thrift decoding > >> > >> I am trying to say that with software engineering only (no change to the > >> format) it is likely possible to increase parquet thrift metadata > parsing > >> speed by 4x. > >> > >> This is not 25x of course, but 4x is non trivial. > >> > >> The fact that no one yet has bothered to invest the time to get the 4x > yet > >> in open source implementations of parquet suggests to me that the > parsing > >> time may not be as critical an issue as we think > >> > >> Andrew > >> > >> On Thu, Aug 15, 2024 at 6:50 AM Alkis Evlogimenos > >> <alkis.evlogime...@databricks.com.invalid> wrote: > >> > >>> The difference in parsing speed between thrift and flatbuffer is >25x. > >>> Thrift has some fundamental design decisions that make decoding slow: > >>> 1. the thrift compact protocol is very data dependent: uleb encoding > for > >>> integers, field ids are deltas from previous. The data dependencies > >>> disallow pipelining of modern cpus > >>> 2. object model does not have a way to use arenas to avoid many > >> allocations > >>> of objects > >>> If we keep thrift, we can potentially get 2 fixed, but fixing 1 > requires > >>> changes to the thrift serialization protocol. Such a change is not > >>> different from switching serialization format. > >>> > >>> > >>> On Thu, Aug 15, 2024 at 12:30 PM Andrew Lamb <andrewlam...@gmail.com> > >>> wrote: > >>> > >>>> I wanted to share some work Xiangpeng Hao did at InfluxData this > summer > >>> on > >>>> the current (thrift) metadata format[1]. > >>>> > >>>> We found that with careful software engineering, we could likely > >> improve > >>>> the speed of reading existing parquet footer format by a factor of 4 > or > >>>> more ([2] contains some specific ideas). While we analyzed the > >>>> Rust implementation, I believe a similar conclusion applies to C/C++. > >>>> > >>>> I realize that there are certain features that switching to an > entirely > >>> new > >>>> footer format would achieve, but the cost to adopting a new format > >>>> across the ecosystem is immense (e.g. Parquet "version 2.0" etc). > >>>> > >>>> It is my opinion that investing the same effort in software > >> optimization > >>>> that would be required for a new footer format would have a much > bigger > >>>> impact > >>>> > >>>> Andrew > >>>> > >>>> [1]: https://www.influxdata.com/blog/how-good-parquet-wide-tables/ > >>>> [2]: https://github.com/apache/arrow-rs/issues/5853 > >>>> > >>>> On Thu, Aug 15, 2024 at 4:26 AM Alkis Evlogimenos > >>>> <alkis.evlogime...@databricks.com.invalid> wrote: > >>>> > >>>>> Hi Julien. > >>>>> > >>>>> Thank you for reconnecting the threads. > >>>>> > >>>>> I have broken down my experiments in a narrative, commit by commit on > >>> how > >>>>> we can go from flatbuffers being ~2x larger than thrift to being > >>> smaller > >>>>> (and at times even half) the size of thrift. This is still on an > >>> internal > >>>>> branch, I will resume work towards the end of this month to port it > >> to > >>>>> arrow so that folks can look at the progress and share ideas. > >>>>> > >>>>> On the benchmarking front I need to build and share a binary for > >> third > >>>>> parties to donate their footers for analysis. > >>>>> > >>>>> The PR for parquet extensions has gotten a few rounds of reviews: > >>>>> https://github.com/apache/parquet-format/pull/254. I hope it will be > >>>>> merged > >>>>> soon. > >>>>> > >>>>> I missed the sync yesterday - for some reason I didn't receive an > >>>>> invitation. Julien could you add me again to the invite list? > >>>>> > >>>>> On Thu, Aug 15, 2024 at 1:32 AM Julien Le Dem <jul...@apache.org> > >>> wrote: > >>>>>> This came up in the sync today. > >>>>>> > >>>>>> There are a few concurrent experiments with flatbuffers for a > >> future > >>>>>> Parquet footer replacement. In itself it is fine and just wanted to > >>>>>> reconnect the threads here so that folks are aware of each other > >> and > >>>> can > >>>>>> share findings. > >>>>>> > >>>>>> - Neelaksh benchmarking and experiments: > >>>>>> > >>>>>> > >> > https://medium.com/@neelaksh-singh/benchmarking-apache-parquet-my-mid-program-journey-as-an-mlh-fellow-bc0b8332c3b1 > >>>>>> https://github.com/Neelaksh-Singh/gresearch_parquet_benchmarking > >>>>>> > >>>>>> - Alkis has also been experimenting and led the proposal for > >> enabling > >>>>>> extending the existing footer. > >>>>>> > >>>>>> > >> > https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit#heading=h.15ohoov5qqm6 > >>>>>> - Xuwei also shared that he is looking into this. > >>>>>> > >>>>>> I would suggest that you all reply to this thread sharing your > >>> current > >>>>>> progress or ideas and a link to your respective repos for > >>>> experimenting. > >>>>>> Best > >>>>>> Julien > >>>>>> >
