irodriguezclaveria commented on PR #6426: URL: https://github.com/apache/paimon/pull/6426#issuecomment-4995626353
Hi @JingsongLi , You're right — real compaction doesn't shrink the HASH index today. DynamicBucketIndexMaintainer only ever calls add() on the hash set, and compact() in AbstractFileStoreWrite doesn't even have a reference to it. So there's no real path where compaction frees up a bucket in the current code, and the existing tests only prove it works if you manually commit a smaller file — not something that happens in practice. New plan: instead of trying to make compaction shrink the HASH index, we're adding a separate, lightweight structure that tracks live primary keys per bucket. It's updated right in notifyNewRecord, using the record's RowKind: insert/update adds the key, delete removes it. This gives us an always-accurate "how many live keys does this bucket have" count without touching the HASH index or waiting for compaction at all. This extra structure only exists when the bucket-refresh feature is enabled (minEmptyBucketsBeforeAsyncCheck != -1). If refresh is off, we skip it entirely — no extra cost for users not using this feature. This also fixes the other issues you raised, since they're all symptoms of relying on an async disk re-scan: - No more double-counting, since there's no more diskRowCountAtLastRefresh delta to reconcile — the in-memory counter is the single source of truth. - No more missed refresh when all non-full buckets disappear — there's no refresh trigger needed anymore, freed buckets are just visible directly in memory. - No more buckets silently dropped when they vanish from the HASH manifest — we don't depend on the manifest at all. - No more async refresh/backpressure complexity — no disk scans, no thread pool involved. Let me know if this direction works for you before we start implementing. Thanks!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
