[
https://issues.apache.org/jira/browse/BEAM-5355?focusedWorklogId=151230&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151230
]
ASF GitHub Bot logged work on BEAM-5355:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Oct/18 15:34
Start Date: 04/Oct/18 15:34
Worklog Time Spent: 10m
Work Description: lgajowy opened a new pull request #6571: [BEAM-5355]
Add GroupByKeyLoadTest
URL: https://github.com/apache/beam/pull/6571
This commit adds the test with all neecssary setup required to run
itmanually.
The following GroupByKey load test can be run on all runners using the
following command (example is for flink):
```
./gradlew run -p sdks/java/load-tests -PloadTest.args='--fanout=5
--iterations=1
--sourceOptions={"numRecords":1000,"splitPointFrequencyRecords":1,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456,"bundleSizeDistribution":{"type":"const","const":42},"forceNumInitialBundles":100,"progressShape":"LINEAR","initializeDelayDistribution":{"type":"const","const":42}}
--stepOptions={"outputRecordsPerInputRecord":2,"preservesInputKeyDistribution":2,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456}}
--runner=FlinkRunner' -Prunner=":beam-runners-flink_2.11"
-PloadTest.mainClass="org.apache.beam.sdk.loadtests.GroupByKeyLoadTest"
```
Runner can be changed by altering the runner-related properties. We can also
add `--flinkMaster=localhost:8081` to run this on a separate flink cluster (not
in memory).
You can skip the syntieticStep pipeline option. Example for dataflow runner:
```
./gradlew run -p sdks/java/load-tests -PloadTest.args='--fanout=5
--iterations=1
--sourceOptions={"numRecords":1000,"splitPointFrequencyRecords":1,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456,"bundleSizeDistribution":{"type":"const","const":42},"forceNumInitialBundles":100,"progressShape":"LINEAR","initializeDelayDistribution":{"type":"const","const":42}}
--runner=DataflowRunner' -Prunner=":beam-runners-google-cloud-dataflow-java"
-PloadTest.mainClass="org.apache.beam.loadtests.GroupByKeyLoadTest"
```
@pabloem could you take a look? I know there is a separate PR for this but
it was easier to completely rebuild the pr due to the fact that now it does not
require Perfkit and runs from the main code. I didn't develop any method of
publishing the metrics to BigQuery - this needs a follow up pull request (is it
ok?).
CC: @echauchot
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone
(e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
| --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 151230)
Time Spent: 1h (was: 50m)
> Create GroupByKey load test for Java SDK
> ----------------------------------------
>
> Key: BEAM-5355
> URL: https://issues.apache.org/jira/browse/BEAM-5355
> Project: Beam
> Issue Type: New Feature
> Components: testing
> Reporter: Lukasz Gajowy
> Assignee: Lukasz Gajowy
> Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This is more thoroughly described in this proposal:
> [https://docs.google.com/document/d/1PuIQv4v06eosKKwT76u7S6IP88AnXhTf870Rcj1AHt4/edit?usp=sharing]
>
> In short: this ticket is about implementing the GroupByKeyLoadIT that uses
> SyntheticStep and Synthetic source to create load on the pipeline.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
