Feifan Wang created FLINK-41431:
-----------------------------------
Summary: RocksDB manual compaction native calls can race with
backend disposal
Key: FLINK-41431
URL: https://issues.apache.org/jira/browse/FLINK-41431
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.20.0
Reporter: Feifan Wang
Since FLINK-26050, the RocksDB state backend periodically schedules manual SST
compaction. All native RocksDB access performed by its asynchronous SST scan
and compaction tasks runs on the shared ioExecutor and never acquires a
rocksDBResourceGuard lease, so it can race with
RocksDBKeyedStateBackend#dispose.
h3. Unguarded native calls (sstmerge)
* SST scan: CompactionScheduler#maybeScan -> CompactionTaskProducer#produce ->
RocksDB#getLiveFilesMetaData()
* Compaction task: CompactionTracker#runWithTracking -> RocksDB#getLongProperty
(auto-compaction count), and Compactor#compact ->
ColumnFamilyHandle#getDescriptor() + RocksDB#compactFiles(...); the latter can
run for a long time
* Task stringification: CompactionTask#toString/equals/hashCode delegate to
ColumnFamilyHandle (native getID/getName); maybeScan formats scan results at
TRACE level outside any lease, so this can run after dispose has closed the
handle
h3. Race with dispose
RocksDBKeyedStateBackend#dispose:
# closes sstMergeManager: this sets the tracker's shut-down flag and stops the
scheduled executor (awaiting termination up to 5s), but does not wait for tasks
already running on the shared ioExecutor
# calls rocksDBResourceGuard.close(): manual compaction holds no lease, so this
returns immediately while native calls are in flight
# closes the column family handles and the DB
Any native call still in flight then operates on freed native resources
(use-after-free / SIGSEGV risk). RocksDB documents that it is not safe to close
the DB while another thread issues manual compaction requests [1].
h3. Reachability
This race is reachable in any job with manual compaction enabled
(state.backend.rocksdb.manual-compaction.min-interval > 0; disabled by default)
when disposal - due to failover, cancellation, rescaling, or stopping with a
savepoint - occurs while a scan or compaction round is in flight. A round's
duration scales with the number and size of candidate SST files, which widens
the window.
We detected this by code inspection while running an internal deployment with
manual compaction enabled by default; we have not yet observed a SIGSEGV in
production, but the race is directly reachable (deterministic unit tests can
pin an in-flight task across dispose).
h3. Suggested fix
We have a patch with deterministic regression tests ready and are happy to open
a PR:
* Pass rocksDBResourceGuard into RocksDBManualCompactionManager; acquire a
lease around the SST scan and around each compaction task (including the
tracker's property read), so dispose waits for in-flight work before releasing
native resources. When the guard is already closed, pending tasks abort quietly
and decrement the tracker's pending-compaction count.
* Make CompactionTask stringification/comparison native-free (handle identity
instead of ColumnFamilyHandle#toString/equals/hashCode), since TRACE logging
formats scan results after the lease is released.
This is complementary to FLINK-36186: aborting manual compaction at close
(DisableManualCompaction) could speed up disposal, but any such approach still
needs a guarded wait for in-flight native calls, and it does not cover the scan
path or task stringification.
[1] https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ - "Is it safe to close
RocksDB while another thread is issuing read, write or manual compaction
requests? A: No."
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
