steFaiz commented on PR #8344:
URL: https://github.com/apache/paimon/pull/8344#issuecomment-4828481473
@JingsongLi Thanks for your reply! This design: "Reference to the oldest
file" is also in my original design docs.
I choose current range-based because:
1. range-based approach is more natural and easy to search
2. oldest filename based approach need modifying current
`DataEvolutionFileStoreScan` logic: Now
a. Data files could be filtered by writeCols, if full schema is [a, b,
c, d] and some files only contains [c]
b. For query like `SELECT c from t`, current implementation only need
all files with writeCols = [c]
c. Now we have to keep the oldest file in plan (or we cannot find
related DVs)
This logic seems weird. Like "if delation-vector is enabled, we need to
plan all column files as well as the oldest file".
3. In theory, for compaction without materialize DVs (materialize DVs need
to rebuild global indices), DVs do not need to rewrite for range-based approach
I agree that the oldest filename based approach will reduce code changes.
Which one do the community prefer? range based or the oldest filename based?
I'll modify this PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
