steFaiz opened a new pull request, #8380: URL: https://github.com/apache/paimon/pull/8380
### Purpose The first part of https://github.com/apache/paimon/issues/8322 As discussed in https://github.com/apache/paimon/pull/8344, this PR tries an another way: use the oldest file as the anchor file to lookup DVs. This approach shifts the complicity from core module to engines module e.g. Spark/Flink/Ray Key design: 1. During `DataEvolutionFileStoreScan`, in post filter, keep the anchor file even it's not needed. 2. Still use `_ROW_ID` to filter out deleted rows. (Not returned positions) . Convert the Deletion Files to Map<Range, DeletionVector> in `DataSplit`. This is because: a. Following the `IndexedSplitRecordReader` implementation. b. returned positions is the concept of a single file. For blobs and vectors, many files are concated (for blobs there even may be placeholders and row range holes). Maintaining returned positions across many concated (and maybe uncontagious) files is complicated and break its semantic (returned position should be within a single file) ### Tests ITCases only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
