steFaiz commented on PR #8354:
URL: https://github.com/apache/paimon/pull/8354#issuecomment-4798068083
@JingsongLi Thanks! My concern is that this situation:
```text
L1: sequence = 7
L2: sequence = 8
L0: sequence = 6
```
is not possible in current Lookup-based merge and compaction. In the above
scenario, if there are two level files:
```text
L2: sequence = 8
L0: sequence = 6
```
Now compaction triggered, L0 files are rewritten to L1, the new L1 file will
be the result generated by merging L0 file and current lowest higher level
file: L2, so the generated L1 file will have sequence number 8:
```text
L2: sequence = 8
L1: sequence = 8
L0: sequence = 6
```
The only possible way is concurrent compaction by different jobs. For
exmaple: the intial file is L2:
```text
L2: sequence = 5
```
then concurrent A & B both write two L0 file:
However:
1. writer A only compact all L0 files, generate L1 file
2. writer B compact both L0 and L2 files, generate L2 file
```text
Writer A Writer B
L0: seq = 7 L0: seq = 8
| |
V V
L1: seq = 7 L2: seq=8
```
Then writer A write new L0 file, seq=6
Then some reader read the snapshot, it reads:
```text
L1: sequence = 7
L2: sequence = 8
L0: sequence = 6
```
But at that case, the result is already been broken, either pick L1 or L2
file will only observe partial result. It's incorrect, no concurrent compaction
should happen for lookup-based field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
