Definitely agree that including `_pos` raises questions. If "debugging" is to be understood as figuring out if the column files have gaps -- just checking the row counts is good enough for that. Is there a lot to be gained from figuring out where exactly the gap is occurring?
On Mon, Aug 24, 2026 at 1:57 PM Marco Kroll <[email protected]> wrote: > > Hi all, > > I just saw the agenda [1] for tomorrow's (2026-08-25) sync and want to +1 the > `_pos` column topic. > My understanding is that this column exists for two reasons: > 1. debugging > 2. detect if writers skipped deleted rows > > My take is that using the dense Null filled representation addresses both of > these issues. > It implicitly encodes the position, very much like for deletion vectors and > since all rows need to be present, comparing the row count of the base file > with the column file can be used to verify that all rows were written. > > The main thing to add to the doc would be that the row order must be > identical to the base file. > > Best > Marco > > [1]: > https://docs.google.com/document/d/1Bd7JVzgajA8-DozzeEE24mID_GLuz6iwj0g4TlcVJcs/edit?tab=t.jvm7iiiulf8q#heading=h.rbisiun18esp
