[
https://issues.apache.org/jira/browse/BEAM-4141?focusedWorklogId=94411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94411
]
ASF GitHub Bot logged work on BEAM-4141:
----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Apr/18 01:14
Start Date: 24/Apr/18 01:14
Worklog Time Spent: 10m
Work Description: lostluck commented on a change in pull request #5184:
BEAM-4141: Drain source when user function processing fails.
URL: https://github.com/apache/beam/pull/5184#discussion_r183581015
##########
File path: sdks/go/pkg/beam/core/runtime/harness/datamgr.go
##########
@@ -29,6 +29,15 @@ import (
)
const chunkSize = int(4e6) // Bytes to put in a single gRPC message. Max is
slightly higher.
+const bufElements = 20 // Number of chunks buffered per reader.
+
+// This is a reduced version of the full gRPC interface to help with testing.
+// TODO(wcn): need a compile-time assertion to make sure this stays synced
with what's
Review comment:
<!--new_thread; commit:c99d4fe17f53af898950a67b6b7d816c570ad9aa;
resolved:1-->
If it weren't indirected by the "check in the generated code" step, I'd say
that line 107 already does it, since it should be checked by the compiler at
that point. Where makeDataChannel is called.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 94411)
Time Spent: 50m (was: 40m)
> Data channel deadlocks when user function fails
> -----------------------------------------------
>
> Key: BEAM-4141
> URL: https://issues.apache.org/jira/browse/BEAM-4141
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Affects Versions: Not applicable
> Reporter: Bill Neubauer
> Assignee: Bill Neubauer
> Priority: Major
> Fix For: Not applicable
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> There is a deadlock condition in the data channel code that occurs when a
> user function fails while processing an element. The producer for the data
> channel is continuing to send information across a channel, but the intended
> consumer has stopped listening. Unfortunately, this channel blocks the entire
> data channel, blocking data for any other DoFn that might be running, causing
> the whole worker to deadlock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
