[
https://issues.apache.org/jira/browse/BEAM-4727?focusedWorklogId=118887&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118887
]
ASF GitHub Bot logged work on BEAM-4727:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Jul/18 22:48
Start Date: 03/Jul/18 22:48
Worklog Time Spent: 10m
Work Description: lostluck opened a new pull request #5885: [BEAM-4727]
Improve metric lookup overhead for predeclared metrics
URL: https://github.com/apache/beam/pull/5885
Introduces a performance for predeclared metrics when they're used within a
single ptransform by caching the storage unit associated with them.
Switched away from using sync.Map in favour of maps with locks. sync.Map
doesn't handle high contention very well.
Experimented with changing from a struct key to a string key, but string
construction time was worse than directly using the struct. It remains
unchanged.
Builds on #5884
Run on @lostluck's desktop:
Before:
BenchmarkMetrics/counter_inplace-12 10000000 226
ns/op 80 B/op 2 allocs/op
BenchmarkMetrics/distribution_inplace-12 10000000 231
ns/op 112 B/op 2 allocs/op
BenchmarkMetrics/gauge_inplace-12 5000000 252
ns/op 112 B/op 2 allocs/op
BenchmarkMetrics/counter_predeclared-12 10000000 222
ns/op 80 B/op 2 allocs/op
BenchmarkMetrics/distribution_predeclared-12 10000000
228 ns/op 112 B/op 2 allocs/op
BenchmarkMetrics/gauge_predeclared-12 5000000
247 ns/op 112 B/op 2 allocs/op
After:
BenchmarkMetrics/counter_inplace-12 5000000
243 ns/op 128 B/op 2 allocs/op
BenchmarkMetrics/distribution_inplace-12 5000000
252 ns/op 160 B/op 2 allocs/op
BenchmarkMetrics/gauge_inplace-12 5000000
266 ns/op 160 B/op 2 allocs/op
BenchmarkMetrics/counter_predeclared-12 20000000
74.3 ns/op 16 B/op 1 allocs/op
BenchmarkMetrics/distribution_predeclared-12 20000000
79.6 ns/op 48 B/op 1 allocs/op
BenchmarkMetrics/gauge_predeclared-12 20000000
92.9 ns/op 48 B/op 1 allocs/op
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone
(e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 118887)
Time Spent: 20m (was: 10m)
> Reduce metrics overhead
> -----------------------
>
> Key: BEAM-4727
> URL: https://issues.apache.org/jira/browse/BEAM-4727
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are a few opportunities to avoid metrics overhead.
> First when setting state in the context, we allocate a new one for the stored
> value, per element. This generates a fair amount of objects for the garbage
> collector too. If we retain and re-use contexts within a bundle, we would
> have the opportunity to save on these costs.
> Also, it's possible that we have overhead on the metric updating paths. We
> can possibly do better than the general sync.Map, and avoid the type
> assertion cost for extracting values of known types from the maps.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
