[
https://issues.apache.org/jira/browse/BEAM-4468?focusedWorklogId=202780&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202780
]
ASF GitHub Bot logged work on BEAM-4468:
----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Feb/19 18:22
Start Date: 22/Feb/19 18:22
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #7927: [BEAM-4468]
Cap lifting cache & use random eviction policy.
URL: https://github.com/apache/beam/pull/7927
There are two optimizations for combiners here:
* Capping the LiftedCombiner cache to 2000 elements and when additional keys
are added, randomly evict an old one to the next stage.
* Do the invoker optimizations done to ParDo for the Combine operations,
which occur up to a per Element basis. This is an extension of [BEAM-4726].
This optimization is necessary for *very large bundles* which may happen for
batch jobs. A random eviction policy was better than simply dumping the whole
cache when the cap is hit. This implementation takes advantage of Go's built in
Random Map Iteration to choose evictees. We can probably do better, but it's
not a implementation I'd like the SDK to maintain, so that is deferred until
adding dependencies is simpler.
For the very large Batch jobs, the invocation optimizations become
necessary, to the tune of ~16% of CPU for a job spent building and throwing
away invokers. This is not surprising, but adds complexity to the code somewhat.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
| --- | --- | ---
See [.test-infra/jenkins/README](../.test-infra/jenkins/README.md) for
trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 202780)
Time Spent: 10m
Remaining Estimate: 0h
> Go SDK-Tune in memory pre-combine caching for Lifted Combines.
> --------------------------------------------------------------
>
> Key: BEAM-4468
> URL: https://issues.apache.org/jira/browse/BEAM-4468
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Requires [BEAM-4276|https://issues.apache.org/jira/browse/BEAM-4276] to be
> completed first.
> Additional performance tweaks to the in memory per-key accumulator cache
> during the pre-combine phase of a lifted combine.
> This can include any of
> * capping the number of key-accumulator in the cache, and draining them
> eagerly after x elements seen, or evicting keys by some heuristic after the
> cache has seen X distinct keys.
> * providing a counter on cache size (key/element counts) exposable through
> the metrics interface or another runner standard counter, to permit observing
> the cache's status, especially if it could grow without bound within a bundle.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
