[
https://issues.apache.org/jira/browse/BEAM-10051?focusedWorklogId=438885&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438885
]
ASF GitHub Bot logged work on BEAM-10051:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 29/May/20 18:12
Start Date: 29/May/20 18:12
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11864:
URL: https://github.com/apache/beam/pull/11864#issuecomment-636112344
`Initial data arrives and instruction never creates a reader` and `Initial
data arrives after instruction ends` should eventually allow progress via a
timeout on how long we expect this data to sit there without an active bundle.
Anything on the order of 10s of seconds seems like plenty of time since we
expect the ProcessBundle instruction to be in the control stream already
(albeit possibly stuck in a network buffer or so). The SDK just needs to record
the id in memory and if it ever sees a bundle for such an id, it would fail it
immediately since it had dropped the data already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 438885)
Time Spent: 1h (was: 50m)
> Misordered check WRT closed data readers.
> -----------------------------------------
>
> Key: BEAM-10051
> URL: https://issues.apache.org/jira/browse/BEAM-10051
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: P2
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This check
> https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/datamgr.go#L269
> in it's current position prevents the "normal teardown" that the reader
> expects. This means that readers for instructions that terminate early such
> as due to splitting stay resident in memory and never close.
> In practice this is benign as the buffer would already be closed, but with
> streaming this memory leak would become noticable.
> The fix is to move the check to after the sentinel check, and additionally
> check there for early termination to avoid closing the buffer twice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
