leaves12138 commented on code in PR #8871:
URL: https://github.com/apache/paimon/pull/8871#discussion_r3662055785
##########
paimon-common/src/main/java/org/apache/paimon/lookup/sort/db/LocalKvDb.java:
##########
@@ -771,6 +864,228 @@ private static MemorySlice copyKey(byte[] key) {
return MemorySlice.wrap(Arrays.copyOf(key, key.length));
}
+ private void ensureNoRangeIterator() {
+ if (openRangeIterators > 0) {
+ throw new IllegalStateException(
+ "The database cannot be modified or closed while a range
iterator is open.");
+ }
+ }
+
+ /** Lazy range iterator with newest-version-wins semantics. */
+ public final class RangeIterator
+ implements KeyValueIterator<MemorySlice, MemorySlice>,
AutoCloseable {
+
+ private final LsmLevels.RangeSnapshot snapshot;
+ private final PriorityQueue<RangeSource> sources;
+
+ @Nullable private MemorySlice currentKey;
+ @Nullable private MemorySlice currentValue;
+ private boolean closed;
+
+ private RangeIterator(
+ LsmLevels.RangeSnapshot snapshot,
+ Map<MemorySlice, byte[]> memoryEntries,
+ byte[] fromInclusive,
+ @Nullable MemorySlice toExclusive)
+ throws IOException {
+ this.snapshot = snapshot;
+ this.sources =
+ new PriorityQueue<>(
+ (left, right) -> {
+ int compare =
keyComparator.compare(left.key(), right.key());
+ return compare != 0
+ ? compare
+ : Integer.compare(left.priority(),
right.priority());
+ });
+
+ int priority = 0;
+ advanceAndAdd(new MemoryRangeSource(priority++,
memoryEntries.entrySet().iterator()));
+ for (File file : snapshot.files()) {
+ advanceAndAdd(
+ new SstRangeSource(
+ priority++, getOrCreateReader(file),
fromInclusive, toExclusive));
Review Comment:
`RangeIterator` eagerly calls `getOrCreateReader` for every overlapping SST,
and every reader remains open in the unbounded `readerCache` until database
close or compaction deletion. I reproduced this with 200 L0 SSTs under `ulimit
-n 256`: before constructing the iterator the process had 150 open file
descriptors, and construction failed on SST 92 with `FileNotFoundException: Too
many open files`. A full-range scan over a large bottom level has the same
shape, so the iterator is lazy only at row level. Please avoid opening all
overlapping files up front (for example, iterate non-overlapping level files
sequentially and otherwise bound the active readers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
