Jack McCluskey created BEAM-13212:
-------------------------------------
Summary: [Go SDK] Add support for setting upper memory limit for
side input cache
Key: BEAM-13212
URL: https://issues.apache.org/jira/browse/BEAM-13212
Project: Beam
Issue Type: Improvement
Components: sdk-go
Reporter: Jack McCluskey
This is implementing https://issues.apache.org/jira/browse/BEAM-8297 for the Go
SDK's side input cache. The current implementation reads the entire side input
into memory and keeps track of how many distinct inputs to hold; however, each
side input is not guaranteed to occupy the same amount of memory. An extra
parameter should be created to be passed in to the SideInputCache
([https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/statecache/statecache.go)]
to set a maximum memory limit in place of a maximum element capacity, then add
a hook (see
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/harness/cache_hooks.go)
and corresponding function to the harnessopts package
([https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/harnessopts/cache.go)]
so users can set this value themselves.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
