leaves12138 commented on code in PR #8857:
URL: https://github.com/apache/paimon/pull/8857#discussion_r3656274319
##########
paimon-common/src/main/java/org/apache/paimon/lookup/sort/db/LocalKvDb.java:
##########
@@ -670,20 +774,39 @@ public SimpleLsmKvDb build() {
}
if (cacheManager == null) {
- cacheManager = new CacheManager(MemorySize.ofMebiBytes(8));
+ cacheManager = new CacheManager(MemorySize.ofMebiBytes(8), 0);
}
SortLookupStoreFactory factory =
new SortLookupStoreFactory(
keyComparator, cacheManager, blockSize,
compressOptions);
+ LongFunction<BloomFilter.Builder> bloomFilterBuilderFactory =
+ bloomFilterEnabled
+ ? expectedEntries ->
+ expectedEntries > 0
+ ? BloomFilter.fixedBuilder(
+ expectedEntries,
bloomFilterFpp)
+ :
BloomFilter.dynamicBuilder(bloomFilterFpp)
+ : expectedEntries -> null;
+ ExecutorService executor = null;
+ if (asyncCompact) {
+ executor =
+ compactionExecutor == null
+ ? Executors.newSingleThreadExecutor(
Review Comment:
Non-blocking scalability concern: every async `LocalKvDb` built without an
injected executor creates its own single-thread executor.
`ClusteringCompactManagerFactory` creates a manager (and therefore a
`LocalKvDb`) per active partition/bucket, so after the first local compaction
each DB retains one `local-kv-db-compaction` thread until that writer/DB is
closed. A local lifecycle test with 24 DBs that each compacted left 24 such
threads alive.
Could we use a dedicated shared bounded executor at the
`FileStoreWrite`/factory scope, with executor ownership handled outside
`LocalKvDb`, or otherwise cap/reuse these threads? I do not consider this a
correctness blocker, but it may become significant for writers with many active
partitions/buckets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
