Hi Leonid, Thanks for the proposal and for continuing the discussion.
I have a couple of questions: - Do you have any suggestions for how writers could reliably produce aligned row groups? In particular, how would an update writer obtain the base file's row-group boundaries in practice? Would all writers be expected to read the base file footer? Also, is there an easy way to implement this using the current Java Parquet writer APIs? This sounds like a nice feature. - Do readers actually need to know in advance that row groups are aligned? It seems a reader could use essentially the same algorithm for both aligned and unaligned files: seek to the nearest row group, discard any rows prior to the desired starting position, and continue reading from there. If page-level skipping is available, it may even avoid reading some of those unnecessary pages. When the row group already begins at the required offset, the discard step simply becomes a no-op. If that is the case, do we actually need an alignment flag at all? Thanks, Peter Leonid Lygin via dev <[email protected]> ezt írta (időpont: 2026. aug. 12., Sze, 17:06): > Hi all! > > Following up on the Column File representation thread > <https://lists.apache.org/thread/jbh1gbrso5h6l4by9rh9poy2cjjtb8j0>, I'd > like to > fire off a discussion about a possible optimization for Column Updates, > where > supporting writers might decide to write out the Column File aligning all > row > groups with the Base File, enabling supporting readers to do simple > zero-copy > reads. > > *Context* > > The original thread settled on a dense representation (i.e. Column Files > contain > exactly the same amount of rows as Base Files), while allowing > unsynchronized > row-group boundaries. > > I'm proposing adding a separate metadata flag (e.g. > `ColumnFile.containsAlignedRowGroups`) which, when set, indicates that the > row > groups contained in the associated Column File are aligned with the Base > File, > and allows readers to directly swap a Base File column chunk with the > Column > File one on byte level. > > By "aligned" here I mean that if a Base File contains two row groups with > rows > e.g. 0-1000 and 1001-2000, the Column File contains two row groups with > exactly > the same row boundaries 0-1000 and 1001-2000; while "unaligned" allows the > Column File to contain row boundaries e.g. 0-1500 and 1501-2000. > > *Performance benefits* > > Assuming a somewhat uniform distrbution of row group byte sizes (say, RG > is the > average row group size in bytes), reading a single Column File incurs > anywhere > from 0 (row groups already match by chance) to 2*RG (one row before, one > row > after) bytes of overhead I/O and memory that is being discarded by the > reader. > > Allowing always aligning row group boundaries allows readers to lower that > overhead cost to exactly zero. > > *Pros:* > > - Writers are not required to implement -- keeping the flag false is > still > valid for any data shape. > - Readers are not required to implement -- aligning row groups doesn't > break > readers that still want to do stitching on row level. > - One less copy of data along the way from parquet to executor. > - Less reader I/O. > > *Cons:* > > - Implementations are optional -- leads to both diverging features in > different implementations on one hand, and diverging codepaths > within a > single Iceberg implementation as support for the unaligned case is > still > required. > - One more field in the Column File struct -- more support work. > > Thanks! > Leonid. >
