Hey Sreeram, I think it's worthwhile to consider what value would be added by tracking the footer size in metadata, but there are other options to address these optimization use cases.
For example, if you take a look at the RangeReadable <https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/io/RangeReadable.java#L68> interface for FileIO implementations, there's a readTail method so that you can optimistically read from the tail end of the file to try to fetch the full footer in a single read. This is even optimized in some of the implementations (like S3InputStream) to leverage backward reads as opposed to seek operations which might have overhead. Depending on the size of the file, you may want to load just the tail or the whole file to avoid all reads. Having the exact value definitely will make this more exact, but I feel like using the above approach can approximate the same performance benefits. Just a thought, -Dan On Tue, Jan 21, 2025 at 12:17 PM Sreeram Garlapati <gsreeramku...@gmail.com> wrote: > Hello Team! > > This is a small improvement proposal to store the *parquet footer size* > as part of the *data_file* metadata in the iceberg manifest > <https://iceberg.apache.org/spec/#manifests>. > *manifest_entry > (2) data_file > (146 Optional) > footer_size_in_bytes* > > *Motivation*: > > - We have several sub-second read use cases on iceberg tables. We > store icebergs and parquets on S3. Every hop to S3 is v.expensive (P99 of > >200 milliseconds). Hence we are trying to see if we can optimize by > cutting down any of these hops. One such hop is during the Parquet file > read., the first read to the parquet, which is to read the last 8 bytes - > to read the - footer size and par1 sequence. > - Iceberg metadata already includes the file_size_in_bytes. Including > the footer size benefits all the readers. ie., readers can directly issue 1 > I/O call to read the footer - *read_parquet_footer(filehandle, > offset=file_size_in_bytes-footer_size_in_bytes-1)* > - This is similar to what we have in the iceberg specification in the > case of storing Table statistics > <https://iceberg.apache.org/spec/#table-statistics>, puffins > > *file-footer-size-in-bytes*. > - This can be easily extended to ORC as needed too. Perhaps, in the > ORC case, an additional property to store the postscript length is also > needed. > > Truly appreciate your thoughts, > Sreeram <https://www.linkedin.com/in/sreeramgarlapati> > >
