zhf999 commented on PR #196: URL: https://github.com/apache/paimon-cpp/pull/196#issuecomment-5264585223
Thanks for your contribution! The feature is gated on every file having a first_row_id, but the global row id isn't actually used — it's added and immediately subtracted back: - probe: global = first_row_id + file_row_id - payload: ToFileSelection translates it right back (from - start, start = first_row_id) What reaches the format reader is a file-local selection bitmap, and the payload reader is already created per file. So the file-local file_row_id from GetPreviousBatchFileRowId is enough on its own; the global round-trip only exists to reuse the existing global ranges pushdown. The downside: it unnecessarily restricts an otherwise general optimization to row-tracking append tables (the only files with a first_row_id). Proposal: implement this purely on top of FileBatchReader, pushing a file-local selection down directly from the probe-collected file_row_ids — no first_row_id or global-row-id dependency. Correctness already relies only on ascending file-local ordering. If global ids are needed for a planned cross-file use, a comment noting that would help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
