[
https://issues.apache.org/jira/browse/BEAM-4468?focusedWorklogId=202968&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202968
]
ASF GitHub Bot logged work on BEAM-4468:
----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Feb/19 02:35
Start Date: 23/Feb/19 02:35
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #7927: [BEAM-4468]
Go SDK Combiner Lifting cache cap & eviction.
URL: https://github.com/apache/beam/pull/7927#discussion_r259561556
##########
File path: sdks/go/pkg/beam/core/runtime/exec/combine.go
##########
@@ -318,6 +318,27 @@ func (n *LiftedCombine) ProcessElement(ctx
context.Context, value *FullValue, va
return n.fail(err)
}
+ // TODO(BEAM-4468): replace with some better implementation
+ // once adding dependencies is easier.
+ // Arbitrary limit until a broader improvement can be demonstrated.
+ const cacheMax = 2000
+ if len(n.cache) > cacheMax {
Review comment:
The 2000 number is also arbitrary, I'm more likely to allow it to be tuned
by the runner/user in some way as it seems a touch low for my usecase. Either
way it's significantly better than run-away growth.
There are pathological cases to every strategy and in practice the best way
to handle that is to detect which case would be better and swap to it when
you're most confident.
In this case I'm referencing the [Iocaine Powder/Scicillian reasoning
strategy for playing Rock Paper Scissors](http://fob.po8.org/node/338), though
with map evictions there are few more moves.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 202968)
Time Spent: 2.5h (was: 2h 20m)
> Go SDK-Tune in memory pre-combine caching for Lifted Combines.
> --------------------------------------------------------------
>
> Key: BEAM-4468
> URL: https://issues.apache.org/jira/browse/BEAM-4468
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: Minor
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Requires [BEAM-4276|https://issues.apache.org/jira/browse/BEAM-4276] to be
> completed first.
> Additional performance tweaks to the in memory per-key accumulator cache
> during the pre-combine phase of a lifted combine.
> This can include any of
> * capping the number of key-accumulator in the cache, and draining them
> eagerly after x elements seen, or evicting keys by some heuristic after the
> cache has seen X distinct keys.
> * providing a counter on cache size (key/element counts) exposable through
> the metrics interface or another runner standard counter, to permit observing
> the cache's status, especially if it could grow without bound within a bundle.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
