Hi, I am the author of the blog here! Happy to answer any questions.
There are a couple of parts, one is regarding relative pointers and a second is the row group chunking system (which for performance purposes could benefit from being implemented in the C/C++ layer). I am happy to help where I can with the latter as that can be done with the current Parquet version too. Thanks, Yucheng On 2024/10/09 15:46:01 Julien Le Dem wrote: > I recommended to them that they join the dev list. I think that's the > easiest to discuss. > IMO, it's a good goal to have relative pointers in the metadata so that a > row group doesn't depend on where it is in a file. > It looks like some aspects of making the data updates more incremental > could leverage Iceberg. > > On Wed, Oct 9, 2024 at 6:52 AM Antoine Pitrou <an...@python.org> wrote: > > > > > I have a contact at Hugging Face who actually notified me of the blog > > post. I can transmit any questions or suggestions if desired. > > > > Regards > > > > Antoine. > > > > > > On Wed, 9 Oct 2024 11:50:51 +0100 > > Steve Loughran > > <st...@cloudera.com.INVALID> wrote: > > > flatbuffer would be the obvious place that would be no compatibility > > issues > > > with existing readers. > > > > > > Also: that looks like a large amount of information to capture statistics > > > on. Has anyone approached them yet? > > > > > > On Wed, 9 Oct 2024 at 03:39, Gang Wu < > > ustcwg-re5jqeeqqe8avxtiumw...@public.gmane.org> wrote: > > > > > > > Thanks Antoine for sharing the blog post! > > > > > > > > I skimmed it quickly and it seems that the main issue is the absolute > > > > file offset used by metadata of page and column chunk. It may take a > > > > long time to migrate if we want to replace them with relative offsets > > > > in the current thrift definition. Perhaps it is a good chance to > > improve > > > > this with the current flatbuffer experiment? > > > > > > > > Best, > > > > Gang > > > > > > > > On Wed, Oct 9, 2024 at 8:51 AM Antoine Pitrou < > > antoine-+zn9apsxkcednm+yrof...@public.gmane.org> wrote: > > > > > > > > > > > > > > Hello, > > > > > > > > > > The Hugging Face developers published this insightful blog post about > > > > > their attempts to deduplicate Parquet files when they have similar > > > > > contents. They offer a couple suggestions for improvement at the end: > > > > > https://huggingface.co/blog/improve_parquet_dedupe > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
