Andrey created FLINK-6412: ----------------------------- Summary: Stream has already been closed durign job cancel Key: FLINK-6412 URL: https://issues.apache.org/jira/browse/FLINK-6412 Project: Flink Issue Type: Bug Affects Versions: 1.2.0 Reporter: Andrey
Steps to reproduce: * create job with RocksDBStateBackend: env.setStateBackend(new RocksDBStateBackend(checkpointRoot)); * run job * cancel job from the Web UI. In logs: {code} 2017-04-28 10:25:57,475 INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source (1/2) (05a3888ed2a232f234a10775826395a0) switched from DEPLOYING to RUNNING. [Source: Custom Source (1/2)] 2017-04-28 10:25:57,476 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - Using user-defined state backend: RocksDB State Backend {isInitialized=false, configuredDbBasePaths=null, initializedDbBasePaths=null, checkpointStreamBackend=File State Backend @ file:/flink/checkpoints/flinktests}. [Map (2/2)] 2017-04-28 10:25:57,476 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - Using user-defined state backend: RocksDB State Backend {isInitialized=false, configuredDbBasePaths=null, initializedDbBasePaths=null, checkpointStreamBackend=File State Backend @ file:/flink/checkpoints/flinktests}. [Source: Custom Source (1/2)] ... 2017-04-28 10:26:29,793 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Source: Custom Source (1/2) (05a3888ed2a232f234a10775826395a0). [flink-akka.actor.default-dispatcher-2] 2017-04-28 10:26:29,794 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to cancel task Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2] 2017-04-28 10:26:29,794 INFO org.apache.flink.runtime.taskmanager.Task - Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e) switched from RUNNING to CANCELING. [flink-akka.actor.default-dispatcher-2] 2017-04-28 10:26:29,796 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2] 2017-04-28 10:26:29,796 INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source (1/2) (05a3888ed2a232f234a10775826395a0) switched from CANCELING to CANCELED. [Source: Custom Source (1/2)] 2017-04-28 10:26:29,797 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Custom Source (1/2) (05a3888ed2a232f234a10775826395a0). [Source: Custom Source (1/2)] 2017-04-28 10:26:29,798 INFO org.apache.flink.core.fs.FileSystem - Ensuring all FileSystem streams are closed for Source: Custom Source (1/2) [Source: Custom Source (1/2)] 2017-04-28 10:26:29,803 INFO org.apache.flink.runtime.taskmanager.TaskManager - Un-registering task and sending final execution state CANCELED to JobManager for task Source: Custom Source (05a3888ed2a232f234a10775826395a0) [flink-akka.actor.default-dispatcher-2] 2017-04-28 10:26:39,608 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to fail task externally Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e). [pool-14-thread-1] 2017-04-28 10:26:39,608 WARN org.apache.flink.streaming.runtime.tasks.StreamTask - Could not properly clean up the async checkpoint runnable. [Canceler for Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e).] java.lang.Exception: Could not properly cancel managed keyed state future. at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91) at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1010) at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.close(StreamTask.java:995) at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:262) at org.apache.flink.util.IOUtils.closeAllQuietly(IOUtils.java:251) at org.apache.flink.util.AbstractCloseableRegistry.close(AbstractCloseableRegistry.java:91) at org.apache.flink.streaming.runtime.tasks.StreamTask.cancel(StreamTask.java:364) at org.apache.flink.runtime.taskmanager.Task$TaskCanceler.run(Task.java:1390) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Stream has already been closed and discarded. at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40) at org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:79) at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:89) ... 8 more Caused by: java.io.IOException: Stream has already been closed and discarded. at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.closeAndGetHandle(FsCheckpointStreamFactory.java:342) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBSnapshotOperation.closeSnapshotStreamAndGetHandle(RocksDBKeyedStateBackend.java:580) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBSnapshotOperation.closeCheckpointStream(RocksDBKeyedStateBackend.java:410) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$1.performOperation(RocksDBKeyedStateBackend.java:298) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$1.performOperation(RocksDBKeyedStateBackend.java:277) at org.apache.flink.runtime.io.async.AbstractAsyncIOCallable.call(AbstractAsyncIOCallable.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:37) at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:915) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more 2017-04-28 10:26:39,608 INFO org.apache.flink.runtime.taskmanager.Task - Task Map (2/2) is already in state CANCELING [pool-14-thread-1] {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)