[
https://issues.apache.org/jira/browse/BEAM-12144?focusedWorklogId=612738&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612738
]
ASF GitHub Bot logged work on BEAM-12144:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Jun/21 15:48
Start Date: 21/Jun/21 15:48
Worklog Time Spent: 10m
Work Description: reuvenlax merged pull request #14714:
URL: https://github.com/apache/beam/pull/14714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 612738)
Time Spent: 1h 40m (was: 1.5h)
> Dataflow streaming worker stuck and unable to get work from Streaming Engine
> ----------------------------------------------------------------------------
>
> Key: BEAM-12144
> URL: https://issues.apache.org/jira/browse/BEAM-12144
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.26.0
> Reporter: Sam Whittle
> Assignee: Sam Whittle
> Priority: P2
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Observed in 2.26 but seems like it could affect later versions as well, as
> previous issues addressing similar problems were before 2.26. This seems
> similar to BEAM-9651 but not the deadlock observed there.
> The thread getting work has the following stack:
> --- Threads (1): [Thread[DispatchThread,1,main]] State: WAITING stack: ---
> [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
>
> [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> [email protected]/java.util.concurrent.Phaser$QNode.block(Phaser.java:1127)
>
> [email protected]/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
>
> [email protected]/java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1057)
>
> [email protected]/java.util.concurrent.Phaser.awaitAdvanceInterruptibly(Phaser.java:747)
>
> app//org.apache.beam.runners.dataflow.worker.windmill.DirectStreamObserver.onNext(DirectStreamObserver.java:49)
>
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$AbstractWindmillStream.send(GrpcWindmillServer.java:662)
>
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$GrpcGetWorkStream.onNewStream(GrpcWindmillServer.java:868)
>
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$AbstractWindmillStream.startStream(GrpcWindmillServer.java:677)
>
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$GrpcGetWorkStream.(GrpcWindmillServer.java:860)
>
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$GrpcGetWorkStream.(GrpcWindmillServer.java:843)
>
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.getWorkStream(GrpcWindmillServer.java:543)
>
> app//org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.streamingDispatchLoop(StreamingDataflowWorker.java:1047)
>
> app//org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$1.run(StreamingDataflowWorker.java:670)
> [email protected]/java.lang.Thread.run(Thread.java:834)
> The status page shows:
> GetWorkStream: 0 buffers, 400 inflight messages allowed, 67108864 inflight
> bytes allowed, current stream is 61355396ms old, last send 61355396ms, last
> response -1ms
> Showing that the stream was created 17 hours ago, sent the header message but
> never received a response. With the stack trace it appears that the header
> was never sent but the stream also didn't terminate with a deadline exceed.
> This seems like a grpc issue to not get an error for the stream, however it
> would be safer to not block indefinitely on the Phaser waiting for the send
> and instead throw an exception after 2x the stream deadline for example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
