This came up in the sync today. There are a few concurrent experiments with flatbuffers for a future Parquet footer replacement. In itself it is fine and just wanted to reconnect the threads here so that folks are aware of each other and can share findings.
- Neelaksh benchmarking and experiments: https://medium.com/@neelaksh-singh/benchmarking-apache-parquet-my-mid-program-journey-as-an-mlh-fellow-bc0b8332c3b1 https://github.com/Neelaksh-Singh/gresearch_parquet_benchmarking - Alkis has also been experimenting and led the proposal for enabling extending the existing footer. https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit#heading=h.15ohoov5qqm6 - Xuwei also shared that he is looking into this. I would suggest that you all reply to this thread sharing your current progress or ideas and a link to your respective repos for experimenting. Best Julien
