mohsenrezaeithe opened a new pull request, #29475:
URL: https://github.com/apache/flink/pull/29475
## What is the purpose of the change
Restoring a job that uses the ForSt state backend at a lower parallelism
(e.g. 4 -> 2) with `state.backend.forst.use-ingest-db-restore-mode: true` fails
with `RocksDBException: NotFound` in `Checkpoint.exportColumnFamily`. This
happens when the primary directory is on an S3-like object store (Presto S3,
native S3), where `mkdirs()` is a no-op and a "directory" only exists while
objects exist under it.
`ForStFileSystemTrackingCreatedDirDecorator` works around the missing
directories by remembering paths passed to `mkdirs()` and reporting them from
`exists()`. RocksDB's export creates `<dir>.tmp` and then renames it to
`<dir>`. The decorator only tracked the source path, so after the rename the
destination was reported as missing and the export failed with `NotFound`.
This PR makes the decorator move the tracked existence from the source to
the destination on `rename()`.
## Brief change log
- `ForStFileSystemTrackingCreatedDirDecorator#rename`: when a tracked
directory is renamed successfully, remove the source path from the tracked set
and add the destination path.
- Add `ObjectStoreLikeFileSystem` (test), a local file system with S3-like
directory semantics: `mkdirs()` is a no-op, writes implicitly create the parent
prefix, and a directory exists only while a file exists below it.
- `ForStFileSystemTrackingCreatedDirDecoratorTest`: add rename tests
(`testRenameDirAndCheck`, `testDummyMkdirRenameDirAndCheck`) that mirror
RocksDB's `<dir>.tmp` -> `<dir>` export rename.
- `ForStStateBackendV2Test`: add a `remoteOnObjectStore` parameter so the
scale-down tests (including `testAsyncStateBackendScaleDown`) also run with the
primary directory on the object-store-like file system.
## Verifying this change
This change added tests and can be verified as follows:
- Added rename tests to `ForStFileSystemTrackingCreatedDirDecoratorTest`,
which fail without the fix.
- Extended `ForStStateBackendV2Test` with object-store-like parameter
combinations (ingest-DB restore mode on and off, filesystem and DB-reusable
checkpoint storage, all restore modes). `testAsyncStateBackendScaleDown`
reproduces the `NotFound` failure without the fix.
- Run: `mvn test -pl flink-state-backends/flink-statebackend-forst
-Dtest='ForStFileSystemTrackingCreatedDirDecoratorTest,ForStStateBackendV2Test'`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no. This only fixes a
failing ForSt restore (ingest-DB mode, scale-in, object-store primary dir) and
doesn't change checkpoint, recovery or deployment behavior.
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
