I
wonder if there might be mileage in building off an existing format
instead of defining another flatbuffer based tabular encoding
We *are* building off an existing format. Parquet will stay as it is, its
logical structure and everything. The encoding used for this logical
structure's metadata is just a detail. In contrast, Arrow (Feather) is
missing so many things that Parquet has, so you would need a lot more work
to get to where Parquet is, if you started off with Arrow.
I also wonder if, for example, feather added better encoding
support, we would get something that met all the requirements, without
needing to effectively create a third file format?
I would argue that feather is faaaaar farther away from being a good
storage format than Parquet is. Yes, you could add a lot of stuff to
feather to make it a great storage format, but it would be way more than
you would need to add to Parquet to make it as good. And you would also be
creating de facto a third format if you change that much. That's also why
Feather has little adoption: It's not a good storage format as of now, as
its based on Arrow and Arrow isn't design for that.
One thing that occurs to me from reading a number of the proposals /
investigations in this space is how similar many of them end up looking
to Apache Arrow Feather / IPC.
I would argue that Parquet and Arrow are actually not that similar. They
are similar, but only "trivially so" (i.e., in ways in which each storage
format would end up at a roughly similar design):
Obvious similarities:
* Most modern formats are chunked columnar, as this is just obviously a
good idea for many scenarios, as you can choose to skip columns and
chunking helps with being able to write without having to buffer all the
file before writing it. In that regard Parquet and Arrow are similar, but
so are ORC and others.
* Formats have some kind of metadata structure describing the file. In
that regard, Parquet and Arrow and all others are similar, but every format
needs this, so the similarity is trivial.
* Both have encodings. Again, any format needs this. Also the fact that
there is overlap in the encodings is just because some encodings are an
obviously good idea (e.g., dictionary).
* Both have a footer. Any format that wants to be able to be written in a
streaming fashion needs this, you don't know the metadata of the file yet
when you start writing, so you cannot use a header and putting the metadata
somewhere in the middle would make it hard to discover.
Now, the metadata structure needs to be encoded, somehow. For this, modern
formats use encoding standards like thrift or flatbuffers. But I would
argue that the actual encoding of the metadata is *not* what defines a
format, at all; it is just a means to encode the logical structure, and
while this structure has some resemblance between Arrow and Parquet, there
are actually a lot of differences. Each format somehow has to do it and
just because two formats may decide to use the same encoding (e.g.,
flatbuffers) that doesn't make them very similar.
But here is the greatest dissimilarity:
Arrow was made to be an IPC streaming format. As such, it needs to be very
fast to read and write. Then the python guys came and invented feather to
make the whole thing also work somewhat as a storage format, but all they
did was basically add a footer; the format itself (i.e., the encodings) are
not at all optimized for a storage format. A storage format may / should
use more heavyweight encodings and it is okay if it is more costly to
write, as you expect it to be read multiple times while an IPC format is
usually only read once.
Could Arrow become more of a storage format by adding more heavyweight
encodings, statistics, and indexes? It sure could! And maybe it will, but
that's not what it was designed for, so I doubt that such things will be a
high priority for Arrow. But I wouldn't - at least as of now - want to base
a storage format on Arrow. There is just too much missing. A lot of things
that Parquet has that Arrow hasn't (all the statistics, bloom filters,
indexes, etc.). It may look like Parquet is becoming Arrow if it adopts
flatbuffers as a metadata format, but I would argue that is not at all the
case. The metadata encoding is just a tiny detail and a lot of other things
are what makes Parquet Parquet.
Cheers,
Jan
Am Do., 15. Aug. 2024 um 15:20 Uhr schrieb Raphael Taylor-Davies
<r.taylordav...@googlemail.com.invalid>:
Hi,
One thing that occurs to me from reading a number of the proposals /
investigations in this space is how similar many of them end up looking
to Apache Arrow Feather / IPC. Parquet does have a narrower type system
and broader encoding and statistics support, but as far as the
underlying file structure is concerned feather looks a lot like many of
the parquet v3 proposals.
Despite this feather has not seen especially broad adoption as far as I
am aware, and I therefore wonder if there might be relevant learnings
here? I also wonder if, for example, feather added better encoding
support, we would get something that met all the requirements, without
needing to effectively create a third file format?
To be clear I have no particular affection for feather, in fact I find
the way it handles dictionaries to be especially distasteful, but I
wonder if there might be mileage in building off an existing format
instead of defining another flatbuffer based tabular encoding...
Kind Regards,
Raphael
On 15/08/2024 13:41, Jan Finis wrote:
I guess most close source implementations have done these optimizations
already, it has just not been done in the open source versions. E.g., we
switched to a custom-built thrift runtime using pool allocators and
string
views instead of copied strings a few years ago, seeing comparable
speed-ups. The C++ thrift library is just horribly inefficient.
I agree with Alkis though that there are some gains that can be
achieved by
optimizing, but the format has inherent drawbacks. Flatbuffers is indeed
more efficient but at the cost of increased size.
Alkis, can you elaborate how you brought the size of Flatbuffers down?
Cheers,
Jan
Am Do., 15. Aug. 2024 um 13:50 Uhr schrieb Andrew Lamb <
andrewlam...@gmail.com>:
I don't disagree that flatbuffers would be faster than thrift decoding
I am trying to say that with software engineering only (no change to
the
format) it is likely possible to increase parquet thrift metadata
parsing
speed by 4x.
This is not 25x of course, but 4x is non trivial.
The fact that no one yet has bothered to invest the time to get the 4x
yet
in open source implementations of parquet suggests to me that the
parsing
time may not be as critical an issue as we think
Andrew
On Thu, Aug 15, 2024 at 6:50 AM Alkis Evlogimenos
<alkis.evlogime...@databricks.com.invalid> wrote:
The difference in parsing speed between thrift and flatbuffer is >25x.
Thrift has some fundamental design decisions that make decoding slow:
1. the thrift compact protocol is very data dependent: uleb encoding
for
integers, field ids are deltas from previous. The data dependencies
disallow pipelining of modern cpus
2. object model does not have a way to use arenas to avoid many
allocations
of objects
If we keep thrift, we can potentially get 2 fixed, but fixing 1
requires
changes to the thrift serialization protocol. Such a change is not
different from switching serialization format.
On Thu, Aug 15, 2024 at 12:30 PM Andrew Lamb <andrewlam...@gmail.com>
wrote:
I wanted to share some work Xiangpeng Hao did at InfluxData this
summer
on
the current (thrift) metadata format[1].
We found that with careful software engineering, we could likely
improve
the speed of reading existing parquet footer format by a factor of 4
or
more ([2] contains some specific ideas). While we analyzed the
Rust implementation, I believe a similar conclusion applies to C/C++.
I realize that there are certain features that switching to an
entirely
new
footer format would achieve, but the cost to adopting a new format
across the ecosystem is immense (e.g. Parquet "version 2.0" etc).
It is my opinion that investing the same effort in software
optimization
that would be required for a new footer format would have a much
bigger
impact
Andrew
[1]: https://www.influxdata.com/blog/how-good-parquet-wide-tables/
[2]: https://github.com/apache/arrow-rs/issues/5853
On Thu, Aug 15, 2024 at 4:26 AM Alkis Evlogimenos
<alkis.evlogime...@databricks.com.invalid> wrote:
Hi Julien.
Thank you for reconnecting the threads.
I have broken down my experiments in a narrative, commit by commit
on
how
we can go from flatbuffers being ~2x larger than thrift to being
smaller
(and at times even half) the size of thrift. This is still on an
internal
branch, I will resume work towards the end of this month to port it
to
arrow so that folks can look at the progress and share ideas.
On the benchmarking front I need to build and share a binary for
third
parties to donate their footers for analysis.
The PR for parquet extensions has gotten a few rounds of reviews:
https://github.com/apache/parquet-format/pull/254. I hope it will
be
merged
soon.
I missed the sync yesterday - for some reason I didn't receive an
invitation. Julien could you add me again to the invite list?
On Thu, Aug 15, 2024 at 1:32 AM Julien Le Dem <jul...@apache.org>
wrote:
This came up in the sync today.
There are a few concurrent experiments with flatbuffers for a
future
Parquet footer replacement. In itself it is fine and just wanted to
reconnect the threads here so that folks are aware of each other
and
can
share findings.
- Neelaksh benchmarking and experiments:
https://medium.com/@neelaksh-singh/benchmarking-apache-parquet-my-mid-program-journey-as-an-mlh-fellow-bc0b8332c3b1
https://github.com/Neelaksh-Singh/gresearch_parquet_benchmarking
- Alkis has also been experimenting and led the proposal for
enabling
extending the existing footer.
https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit#heading=h.15ohoov5qqm6
- Xuwei also shared that he is looking into this.
I would suggest that you all reply to this thread sharing your
current
progress or ideas and a link to your respective repos for
experimenting.
Best
Julien
