Thanks for the quick response! My biggest concern with `_pos` is not performance but rather clarity and implementation divergence:
1. including `_pos` is redundant, and (at least for me) provokes a re-read of the row alignment section — "why include `_pos` if files are fully aligned?"; 2. having `_pos` fully duplicate the row position, there are two different legal ways to implement reads: either positionally, or using `_pos`. On Wed, Aug 26, 2026 at 2:35 PM Gábor Kaszab <[email protected]> wrote: > > Hey All, > > Thanks for bringing this up! (for me the initial mail went to spam, though...) > > Technically, with the dense representation we don't really need the _pos > column in the column files, unless for troubleshooting. While checking the > row counts is good, if they don't match we might get a better understanding > on what the writer missed writing if we had the _pos col, also the order > could be verified. > > Apart from debugging, I think either way is just fine. An additional detail > to consider is that according to my experiments, there isn't really any > storage cost for writing the _pos with delta encoding (e.g. with Parquet V2). > So the conclusion was that since it comes for free, and might help for > debugging, why not write it. > > Should we reopen this question? Any further feedback is welcome. > > Best Regards, > Gabor > > Leonid Lygin via dev <[email protected]> ezt írta (időpont: 2026. aug. > 26., Sze, 14:10): >> >> Definitely agree that including `_pos` raises questions. >> >> If "debugging" is to be understood as figuring out if the column files >> have gaps -- just checking the row counts is good enough for that. Is >> there a lot to be gained from figuring out where exactly the gap is >> occurring? >> >> On Mon, Aug 24, 2026 at 1:57 PM Marco Kroll >> <[email protected]> wrote: >> > >> > Hi all, >> > >> > I just saw the agenda [1] for tomorrow's (2026-08-25) sync and want to +1 >> > the `_pos` column topic. >> > My understanding is that this column exists for two reasons: >> > 1. debugging >> > 2. detect if writers skipped deleted rows >> > >> > My take is that using the dense Null filled representation addresses both >> > of these issues. >> > It implicitly encodes the position, very much like for deletion vectors >> > and since all rows need to be present, comparing the row count of the base >> > file with the column file can be used to verify that all rows were written. >> > >> > The main thing to add to the doc would be that the row order must be >> > identical to the base file. >> > >> > Best >> > Marco >> > >> > [1]: >> > https://docs.google.com/document/d/1Bd7JVzgajA8-DozzeEE24mID_GLuz6iwj0g4TlcVJcs/edit?tab=t.jvm7iiiulf8q#heading=h.rbisiun18esp
