Hey All, Thanks for bringing this up! (for me the initial mail went to spam, though...)
Technically, with the dense representation we don't really need the _pos column in the column files, unless for troubleshooting. While checking the row counts is good, if they don't match we might get a better understanding on what the writer missed writing if we had the _pos col, also the order could be verified. Apart from debugging, I think either way is just fine. An additional detail to consider is that according to my experiments, there *isn't* really *any storage cost* for writing the _pos with delta encoding (e.g. with Parquet V2). So the conclusion was that since it comes for free, and might help for debugging, why not write it. Should we reopen this question? Any further feedback is welcome. Best Regards, Gabor Leonid Lygin via dev <[email protected]> ezt írta (időpont: 2026. aug. 26., Sze, 14:10): > Definitely agree that including `_pos` raises questions. > > If "debugging" is to be understood as figuring out if the column files > have gaps -- just checking the row counts is good enough for that. Is > there a lot to be gained from figuring out where exactly the gap is > occurring? > > On Mon, Aug 24, 2026 at 1:57 PM Marco Kroll > <[email protected]> wrote: > > > > Hi all, > > > > I just saw the agenda [1] for tomorrow's (2026-08-25) sync and want to > +1 the `_pos` column topic. > > My understanding is that this column exists for two reasons: > > 1. debugging > > 2. detect if writers skipped deleted rows > > > > My take is that using the dense Null filled representation addresses > both of these issues. > > It implicitly encodes the position, very much like for deletion vectors > and since all rows need to be present, comparing the row count of the base > file with the column file can be used to verify that all rows were written. > > > > The main thing to add to the doc would be that the row order must be > identical to the base file. > > > > Best > > Marco > > > > [1]: > https://docs.google.com/document/d/1Bd7JVzgajA8-DozzeEE24mID_GLuz6iwj0g4TlcVJcs/edit?tab=t.jvm7iiiulf8q#heading=h.rbisiun18esp >
