wangyong9999 commented on code in PR #267:
URL: https://github.com/apache/paimon-cpp/pull/267#discussion_r3897107230
##########
src/paimon/common/global_index/btree/btree_global_indexer.cpp:
##########
@@ -57,7 +91,8 @@ Result<std::unique_ptr<BTreeGlobalIndexer>>
BTreeGlobalIndexer::Create(
double high_priority_pool_ratio,
OptionsUtils::GetValueFromMap<double>(options,
BtreeDefs::kBtreeIndexHighPriorityPoolRatio,
BtreeDefs::kDefaultBtreeIndexHighPriorityPoolRatio));
- auto cache_manager = std::make_shared<CacheManager>(cache_size,
high_priority_pool_ratio);
+ std::shared_ptr<CacheManager> cache_manager =
+ GetSharedCacheManager(cache_size, high_priority_pool_ratio);
Review Comment:
Fixed in b35e9c70. Shared page keys now include an internal backend
namespace. Filesystem-backed readers share by FileSystem instance; unknown
readers are isolated by reader identity, and cached values retain the namespace
owner until eviction.
##########
src/paimon/common/global_index/btree/btree_global_indexer.cpp:
##########
@@ -42,9 +45,40 @@
#include "paimon/executor.h"
#include "paimon/global_index/bitmap_global_index_result.h"
#include "paimon/memory/bytes.h"
+#include "paimon/memory/memory_pool.h"
#include "paimon/utils/roaring_bitmap64.h"
namespace paimon {
+namespace {
+
+struct SharedCacheManager {
+ SharedCacheManager(int64_t cache_size, double high_priority_pool_ratio)
+ : cache_pool(GetDefaultPool()),
+ cache_manager(std::make_shared<CacheManager>(cache_size,
high_priority_pool_ratio)) {}
+
+ // Keep the allocator alive until after cache_manager releases all cached
pages.
+ std::shared_ptr<MemoryPool> cache_pool;
+ std::shared_ptr<CacheManager> cache_manager;
+};
+
+std::shared_ptr<CacheManager> GetSharedCacheManager(int64_t cache_size,
+ double
high_priority_pool_ratio) {
+ using CacheConfig = std::pair<int64_t, double>;
+ static std::mutex mutex;
+ static std::map<CacheConfig, SharedCacheManager> cache_managers;
Review Comment:
Fixed in 9cd2147b. The process now retains at most four cache configurations
and evicts the least recently used registry entry. An evicted manager survives
only while an active reader still references it.
##########
src/paimon/core/operation/file_store_scan.cpp:
##########
@@ -150,12 +150,12 @@ Result<std::shared_ptr<FileStoreScan::RawPlan>>
FileStoreScan::CreatePlan() cons
snapshot.has_value() && scan_mode_ == ScanMode::ALL &&
core_options_.GetScanManifestEntryCacheMaxSnapshots() > 0 &&
core_options_.GetCache() != nullptr && !table_path_.empty() &&
- !row_range_index_.has_value() && bucket_filter_.has_value();
+ !row_range_index_.has_value();
Review Comment:
Fixed in 9cd2147b. Bucket cache entries now store the snapshot generation
derived from the base and delta manifest-list names. A hit requires both
snapshot ID and generation to match.
##########
src/paimon/core/operation/file_store_scan.cpp:
##########
@@ -347,16 +347,21 @@ Status FileStoreScan::ReadManifestEntriesWithCache(
}
*cache_hit = false;
- // Rebuild the target snapshot bucket from all manifests and write the
live entries back to the
- // cache.
- std::vector<ManifestFileMeta> bucket_manifest_metas;
- for (const auto& meta : all_manifest_metas) {
- if (MayContainBucket(meta, bucket)) {
- bucket_manifest_metas.push_back(meta);
+ if (bucket) {
+ std::vector<ManifestFileMeta> bucket_manifest_metas;
+ for (const auto& meta : all_manifest_metas) {
+ if (MayContainBucket(meta, bucket.value())) {
+ bucket_manifest_metas.push_back(meta);
+ }
}
+ PAIMON_RETURN_NOT_OK(
+ ReadAndMergeBucketFileEntries(bucket_manifest_metas,
bucket.value(), manifest_entries));
+ } else {
+ std::vector<ManifestEntry> unmerged_entries;
+ PAIMON_RETURN_NOT_OK(
+ ReadFileEntries(all_manifest_metas, &unmerged_entries,
/*apply_scan_filter=*/false));
Review Comment:
Fixed in 9cd2147b by removing the no-bucket whole-table manifest cache path.
Only bounded bucket-specific entries remain, so an oversized whole-table value
cannot trigger repeated materialization and failed insertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
