steFaiz commented on PR #8344:
URL: https://github.com/apache/paimon/pull/8344#issuecomment-4808165779
I thought about the design and I think that making row ranged-DV aligned
with normal file ranges is much more simple and easy to manage, compared to
fully decoupled mode. Just like filename-based implementation.
The main logic:
1. At first DV generation, generate DVs aligned with existing normal data
file ranges
2. During compaction, if some data files triggered row-level compaction and
are merge into larger data file range, we merge related DVs too.
This is because it's complicated to update DVs if we let DVs can be fully
decouple, consider this scenario:
Current DV ranges:
```java
// Current DV:
[0, 100] , [101, 200], [230, 300]
```
Update a new DV range [50, 250]
This will involves:
1. among all DVs, binary search to find all overlapping DVs. here we find:
[0, 100] , [101, 200] , [230, 300]
2. check each DV:
a. For [0, 100], it's only partial overlapping with [50, 250], so we may
have to split the new DV into [50, 100], [101, 250]
b. For [101, 200] and [230, 300], we need to merge it with current
splitted new DV: [101, 250], but how to merge them? Merge into a single big
range [101, 300]? or split it into some selected size?
If we don't design sophiscated merge/split mechanism, there might be some
DVs' size too big or too small.
I think we could:
1. At read path: capable of reading any decoupled DV ranges (already
implemented in this PR)
2. At write/compaction path: force DV ranges be aligned with existing normal
file ranges.
@JingsongLi Could you please review this proposal and share your thoughts?
Any feedback would be greatly appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
