leaves12138 commented on code in PR #8872:
URL: https://github.com/apache/paimon/pull/8872#discussion_r3659121571
##########
paimon-common/src/main/java/org/apache/paimon/lookup/sort/db/LocalKvDb.java:
##########
@@ -611,25 +618,34 @@ private SstFileMetadata
writeMemTableToSst(TreeMap<MemorySlice, byte[]> data)
throws IOException {
File sstFile = newSstFile();
SortLookupStoreWriter writer =
- storeFactory.createWriter(sstFile,
bloomFilterBuilderFactory.apply(data.size()));
- MemorySlice minKey = null;
- MemorySlice maxKey = null;
- long tombstoneCount = 0;
+ storeFactory.createWriter(
+ sstFile,
+ bloomFilterBuilderFactory.apply(
+ mergeOperator == null ? data.size() :
UNKNOWN_NUM_ENTRIES));
+ MemorySlice[] boundaryKeys = new MemorySlice[2];
+ long[] tombstoneCount = new long[1];
+ RecordCombiningWriter combiningWriter =
+ new RecordCombiningWriter(
+ mergeOperator,
+ (key, value) -> {
+ writer.put(key.copyBytes(), value);
+ if (boundaryKeys[0] == null) {
+ boundaryKeys[0] = key;
+ }
+ boundaryKeys[1] = key;
+ if (isTombstone(value)) {
+ tombstoneCount[0]++;
+ }
+ });
try {
for (Map.Entry<MemorySlice, byte[]> entry : data.entrySet()) {
- writer.put(entry.getKey().copyBytes(), entry.getValue());
- if (minKey == null) {
- minKey = entry.getKey();
- }
- maxKey = entry.getKey();
- if (isTombstone(entry.getValue())) {
- tombstoneCount++;
- }
+ combiningWriter.put(entry.getKey(), entry.getValue());
Review Comment:
`RecordCombiningWriter` can collapse multiple keys into the first key
without writing tombstones for the consumed keys. If an older run still
contains one of those keys (for example from `bulkLoad`), reads fall through to
the stale value after flush. I reproduced this with old `a-1/a-2`, then new
`a-1/a-2`: `get(a-2)` returns `new-2` before flush and `old-2` after flush.
Partial compaction has the same risk when older runs remain. Please ensure
every consumed key shadows older versions, or restrict combining to cases where
no older run can remain, and add a regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
