snuyanzin commented on PR #23914:
URL: https://github.com/apache/flink/pull/23914#issuecomment-1853554316
@Jiabao-Sun , @XComp
it seems I found the reason
junit5.10.1 makes it always failing and it is becoming a bit more clear
there are 2 threads
1. junit5 trying to delete dir
2. cleanup snapshot
```
at
org.apache.flink.runtime.state.SnapshotDirectory.cleanup(SnapshotDirectory.java:93)
at
org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase$NativeRocksDBSnapshotResources.release(RocksDBSnapshotStrategyBase.java:384)
at
org.apache.flink.runtime.state.SnapshotStrategyRunner$1.cleanupProvidedResources(SnapshotStrategyRunner.java:97)
at
org.apache.flink.runtime.state.AsyncSnapshotCallable.cleanup(AsyncSnapshotCallable.java:163)
at
org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:87)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.flink.util.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:508)
at
org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(OperatorSnapshotFinalizer.java:54)
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.finalizeNonFinishedSnapshots(AsyncCheckpointRunnable.java:191)
at
org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:124)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
suspect that in JUnit5 they made removal in `AfterEach` which makes it
concurrent with checkpoint cleanup...
Since junit 5.10.1 makes it failing even locally
as a WA I replaced `TempDir` with
```java
val baseCheckpointPath =
Files.createTempDirectory(getClass.getCanonicalName)
Files.deleteIfExists(baseCheckpointPath);
```
locally it helps
https://dev.azure.com/snuyanzin/flink/_build/results?buildId=2674&view=results
now it is running on my ci to see whether it helps or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
