This commit adds the test with all neecssary setup required to run itmanually.
The following GroupByKey load test can be run on all runners using the
following command (example is for flink):
```
./gradlew run -p sdks/java/load-tests -PloadTest.args='--fanout=5
--iterations=1
--sourceOptions={"numRecords":1000,"splitPointFrequencyRecords":1,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456,"bundleSizeDistribution":{"type":"const","const":42},"forceNumInitialBundles":100,"progressShape":"LINEAR","initializeDelayDistribution":{"type":"const","const":42}}
--stepOptions={"outputRecordsPerInputRecord":2,"preservesInputKeyDistribution":2,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456}}
--runner=FlinkRunner' -Prunner=":beam-runners-flink_2.11"
-PloadTest.mainClass="org.apache.beam.sdk.loadtests.GroupByKeyLoadTest"
```
Runner can be changed by altering the runner-related properties. We can also
add `--flinkMaster=localhost:8081` to run this on a separate flink cluster (not
in memory).
You can skip the syntieticStep pipeline option. Example for dataflow runner:
```
./gradlew run -p sdks/java/load-tests -PloadTest.args='--fanout=5
--iterations=1
--sourceOptions={"numRecords":1000,"splitPointFrequencyRecords":1,"keySizeBytes":10,"valueSizeBytes":20,"numHotKeys":3,"hotKeyFraction":0.3,"seed":123456,"bundleSizeDistribution":{"type":"const","const":42},"forceNumInitialBundles":100,"progressShape":"LINEAR","initializeDelayDistribution":{"type":"const","const":42}}
--runner=DataflowRunner' -Prunner=":beam-runners-google-cloud-dataflow-java"
-PloadTest.mainClass="org.apache.beam.loadtests.GroupByKeyLoadTest"
```
@pabloem could you take a look? I know there is a separate PR for this but it
was easier to completely rebuild the pr due to the fact that now it does not
require Perfkit and runs from the main code. I didn't develop any method of
publishing the metrics to BigQuery - this needs a follow up pull request (is it
ok?).
CC: @echauchot
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g.
`@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
| --- | --- | ---
[ Full content available at: https://github.com/apache/beam/pull/6571 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
