[
https://issues.apache.org/jira/browse/BEAM-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657535#comment-16657535
]
Micah Wylde commented on BEAM-5797:
-----------------------------------
The relevant stack trace appears to be:
{code:java}
2018-10-19 15:54:52,905 ERROR
org.apache.flink.streaming.runtime.tasks.StreamTask - Error during disposal of
stream operator.
java.lang.RuntimeException: Failed to finish remote bundle
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner.finishBundle(ExecutableStageDoFnOperator.java:241)
at
org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:87)
at
org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:118)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:674)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.dispose(DoFnOperator.java:391)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator.dispose(ExecutableStageDoFnOperator.java:166)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:473)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:374)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Already closed.
at
org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:95)
at
org.apache.beam.runners.fnexecution.control.SdkHarnessClient$ActiveBundle.close(SdkHarnessClient.java:251)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner.finishBundle(ExecutableStageDoFnOperator.java:238)
... 9 more
Suppressed: java.lang.IllegalStateException: Processing bundle failed, TODO:
[BEAM-3962] abort bundle.
at
org.apache.beam.runners.fnexecution.control.SdkHarnessClient$ActiveBundle.close(SdkHarnessClient.java:266)
... 10 more{code}
> SDK workers are not always killed when Flink pipeline finishes
> --------------------------------------------------------------
>
> Key: BEAM-5797
> URL: https://issues.apache.org/jira/browse/BEAM-5797
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Micah Wylde
> Priority: Major
> Labels: portability-flink
>
> Beam python workers are spun up as part of a pipeline execution, and killed
> once that pipeline has been cancelled or failed. However, in some situations
> we see the workers hanging around indefinitely until they are manually killed
> or the taskmanager is restarted. The behavior seems to only occur with
> streaming pipelines, and appears non-deterministic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
