[
https://issues.apache.org/jira/browse/BEAM-13130?focusedWorklogId=670812&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670812
]
ASF GitHub Bot logged work on BEAM-13130:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Oct/21 15:27
Start Date: 27/Oct/21 15:27
Worklog Time Spent: 10m
Work Description: lostluck commented on a change in pull request #15815:
URL: https://github.com/apache/beam/pull/15815#discussion_r737589733
##########
File path: sdks/go/pkg/beam/core/runtime/harness/statemgr.go
##########
@@ -107,10 +105,6 @@ func (s *ScopedStateReader) Close() error {
s.mu.Lock()
s.closed = true
s.mgr = nil
- for _, r := range s.opened {
- r.Close() // force close all opened readers
Review comment:
So this PR isn't ensuring all readers are closed in user panic/error
cases like the flawed earlier approach does. A user panic or a dofn returning
an error will leave readers open, causing other leaks.
Something also needs to change in exec/pardo.go to close any open streams in
that case, using a defer to cover panic handling cases.
Essentially current change only works for happy path cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 670812)
Time Spent: 20m (was: 10m)
> [Go SDK] Side Input Memory Leak
> -------------------------------
>
> Key: BEAM-13130
> URL: https://issues.apache.org/jira/browse/BEAM-13130
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Jack McCluskey
> Assignee: Jack McCluskey
> Priority: P2
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Keeping a list of the open stateKeyReader structs in the ScopedStateReader is
> holding on to memory for the life of bundles, leading to significant memory
> usage in large batch jobs. This list needs to be removed to the garbage
> collector can clean up stateKeyReader structs sooner.
>
> stateKeyReader:
> [https://github.com/apache/beam/blob/d916c1f55e57a61b54135d0922ad8660735bd287/sdks/go/pkg/beam/core/runtime/harness/statemgr.go#L105]
> List kept by ScopedStateReader:
> https://github.com/apache/beam/blob/d916c1f55e57a61b54135d0922ad8660735bd287/sdks/go/pkg/beam/core/runtime/harness/statemgr.go#L39
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
