[
https://issues.apache.org/jira/browse/BEAM-4552?focusedWorklogId=232716&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-232716
]
ASF GitHub Bot logged work on BEAM-4552:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Apr/19 10:30
Start Date: 25/Apr/19 10:30
Worklog Time Spent: 10m
Work Description: aromanenko-dev commented on issue #8387: [DO NOT
MERGE][BEAM-4552] Use Spark AccumulatorsV2 API
URL: https://github.com/apache/beam/pull/8387#issuecomment-486617009
I performed 3 times in a row every Nexmark run against master and my feature
branch with `AccumulatorV2` and I see that for some tests we have always better
results with new accumulator API.
Used command:
```
$ ./gradlew :beam-sdks-java-nexmark:run
-Pnexmark.runner=":beam-runners-spark" -Pnexmark.args="--runner=SparkRunner
--suite=SMOKE --streamTimeout=60 --streaming=false --manageResources=false
--monitorJobs=true"
```
The results of testing:
```
Master branch
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.8 56980.1 100000
0001 0.6 158982.5 92000
0002 0.4 237529.7 351
0003 3.7 26824.0 580
0004 1.5 6738.5 40
0005 4.4 22888.5 12
0006 1.4 7102.3 103
0007 4.1 24230.7 1
0008 4.1 24539.9 6000
0009 1.4 7256.9 298
0010 2.6 37792.9 1
0011 2.4 40933.3 1919
0012 2.4 41841.0 1919
0013 0.8 120772.9 92000
0014 6.4 31060.7 184000
AccumulatorV2 branch:
Performance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results
(Baseline)
0000 1.8 55157.2 100000
0001 0.7 133868.8 92000
0002 0.6 168918.9 351
0003 3.5 28498.1 580
0004 1.2 8650.5 40
0005 3.1 32690.4 12
0006 1.0 9842.5 103
0007 2.8 35868.0 1
0008 3.3 30404.4 6000
0009 1.4 7369.2 298
0010 1.6 62774.6 1
0011 1.7 59101.7 1919
0012 1.8 57012.5 1919
0013 0.5 214592.3 92000
0014 5.7 35026.3 184000
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 232716)
Time Spent: 1h 40m (was: 1.5h)
> Spark runner should use AccumulatorsV2
> --------------------------------------
>
> Key: BEAM-4552
> URL: https://issues.apache.org/jira/browse/BEAM-4552
> Project: Beam
> Issue Type: Sub-task
> Components: runner-spark
> Reporter: Ismaël Mejía
> Assignee: Alexey Romanenko
> Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Spark runner currently uses the old (and deprecated) API for Accumulators for
> Beam metrics. We should upgrade this now that we are fully based on Spark 2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
