[
https://issues.apache.org/jira/browse/FLINK-41431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18142855#comment-18142855
]
Feifan Wang edited comment on FLINK-41431 at 11/11/26, 7:54 AM:
----------------------------------------------------------------
Hi [~roman], would you mind taking a look when you have a moment?
We hit this while running sstmerge with manual compaction enabled by default in
an internal deployment; the patch we have ready follows the existing
ResourceGuard lease pattern (lease around the SST scan and each compaction
task, native-free task stringification) and comes with deterministic regression
tests.
Happy to open a PR if the approach sounds reasonable.
was (Author: feifan wang):
Hi [~roman], would you mind taking a look when you have a moment?
We hit this while running sstmerge with manual compaction enabled by default in
an internal deployment; the patch we have ready follows the existing
ResourceGuard lease pattern (lease around the SST scan and each compaction
task, native-free task stringification) and comes with deterministic regression
tests.
Happy to open a PR if the approach sounds reasonable.
> RocksDB manual compaction native calls can race with backend disposal
> ---------------------------------------------------------------------
>
> Key: FLINK-41431
> URL: https://issues.apache.org/jira/browse/FLINK-41431
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 1.20.0
> Reporter: Feifan Wang
> Priority: Major
>
> Since FLINK-26050, the RocksDB state backend periodically schedules manual
> SST compaction. All native RocksDB access performed by its asynchronous SST
> scan and compaction tasks runs on the shared ioExecutor and never acquires a
> rocksDBResourceGuard lease, so it can race with
> RocksDBKeyedStateBackend#dispose.
> h3. Unguarded native calls (sstmerge)
> * SST scan: CompactionScheduler#maybeScan -> CompactionTaskProducer#produce
> -> RocksDB#getLiveFilesMetaData()
> * Compaction task: CompactionTracker#runWithTracking ->
> RocksDB#getLongProperty (auto-compaction count), and Compactor#compact ->
> ColumnFamilyHandle#getDescriptor() + RocksDB#compactFiles(...); the latter
> can run for a long time
> * Task stringification: CompactionTask#toString/equals/hashCode delegate to
> ColumnFamilyHandle (native getID/getName); maybeScan formats scan results at
> TRACE level outside any lease, so this can run after dispose has closed the
> handle
> h3. Race with dispose
> RocksDBKeyedStateBackend#dispose:
> # closes sstMergeManager: this sets the tracker's shut-down flag and stops
> the scheduled executor (awaiting termination up to 5s), but does not wait for
> tasks already running on the shared ioExecutor
> # calls rocksDBResourceGuard.close(): manual compaction holds no lease, so
> this returns immediately while native calls are in flight
> # closes the column family handles and the DB
> Any native call still in flight then operates on freed native resources
> (use-after-free / SIGSEGV risk). RocksDB documents that it is not safe to
> close the DB while another thread issues manual compaction requests [1].
> h3. Reachability
> This race is reachable in any job with manual compaction enabled
> (state.backend.rocksdb.manual-compaction.min-interval > 0; disabled by
> default) when disposal - due to failover, cancellation, rescaling, or
> stopping with a savepoint - occurs while a scan or compaction round is in
> flight. A round's duration scales with the number and size of candidate SST
> files, which widens the window.
> We detected this by code inspection while running an internal deployment with
> manual compaction enabled by default; we have not yet observed a SIGSEGV in
> production, but the race is directly reachable (deterministic unit tests can
> pin an in-flight task across dispose).
> h3. Suggested fix
> We have a patch with deterministic regression tests ready and are happy to
> open a PR:
> * Pass rocksDBResourceGuard into RocksDBManualCompactionManager; acquire a
> lease around the SST scan and around each compaction task (including the
> tracker's property read), so dispose waits for in-flight work before
> releasing native resources. When the guard is already closed, pending tasks
> abort quietly and decrement the tracker's pending-compaction count.
> * Make CompactionTask stringification/comparison native-free (handle identity
> instead of ColumnFamilyHandle#toString/equals/hashCode), since TRACE logging
> formats scan results after the lease is released.
> This is complementary to FLINK-36186: aborting manual compaction at close
> (DisableManualCompaction) could speed up disposal, but any such approach
> still needs a guarded wait for in-flight native calls, and it does not cover
> the scan path or task stringification.
> [1] https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ - "Is it safe to
> close RocksDB while another thread is issuing read, write or manual
> compaction requests? A: No."
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
