[
https://issues.apache.org/jira/browse/BEAM-13130?focusedWorklogId=670839&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670839
]
ASF GitHub Bot logged work on BEAM-13130:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Oct/21 15:55
Start Date: 27/Oct/21 15:55
Worklog Time Spent: 10m
Work Description: jrmccluskey commented on a change in pull request
#15815:
URL: https://github.com/apache/beam/pull/15815#discussion_r737617649
##########
File path: sdks/go/pkg/beam/core/runtime/harness/statemgr.go
##########
@@ -107,10 +105,6 @@ func (s *ScopedStateReader) Close() error {
s.mu.Lock()
s.closed = true
s.mgr = nil
- for _, r := range s.opened {
- r.Close() // force close all opened readers
Review comment:
The open streams get closed in postInvoke() in the ParDo code and we
could add some extra handling to the fail() method to clear out the side input
streams too (e.g. check if the side input list is non-nil and call Reset() for
them if not.) It's also worth nothing that there currently isn't a deferred
usage for the ScopedStateReader's Close() function that I can find.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 670839)
Time Spent: 0.5h (was: 20m)
> [Go SDK] Side Input Memory Leak
> -------------------------------
>
> Key: BEAM-13130
> URL: https://issues.apache.org/jira/browse/BEAM-13130
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Jack McCluskey
> Assignee: Jack McCluskey
> Priority: P2
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Keeping a list of the open stateKeyReader structs in the ScopedStateReader is
> holding on to memory for the life of bundles, leading to significant memory
> usage in large batch jobs. This list needs to be removed to the garbage
> collector can clean up stateKeyReader structs sooner.
>
> stateKeyReader:
> [https://github.com/apache/beam/blob/d916c1f55e57a61b54135d0922ad8660735bd287/sdks/go/pkg/beam/core/runtime/harness/statemgr.go#L105]
> List kept by ScopedStateReader:
> https://github.com/apache/beam/blob/d916c1f55e57a61b54135d0922ad8660735bd287/sdks/go/pkg/beam/core/runtime/harness/statemgr.go#L39
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
