[
https://issues.apache.org/jira/browse/BEAM-9436?focusedWorklogId=415121&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415121
]
ASF GitHub Bot logged work on BEAM-9436:
----------------------------------------
Author: ASF GitHub Bot
Created on: 02/Apr/20 23:11
Start Date: 02/Apr/20 23:11
Worklog Time Spent: 10m
Work Description: iemejia commented on issue #11055: [BEAM-9436] Improve
GBK in spark structured streaming runner
URL: https://github.com/apache/beam/pull/11055#issuecomment-608138463
Code looks ok. I could not consistently reproduce the results, when I run
the load test in the branch vs master (with the gradleRun patch to include
spark-sql) I see results that vary a lot, RAM compsumption is almost equal, but
runtime tends to be faster but mostly by a tiny difference on this branch but
some runs curiously take 4 more times than others (in both branches), so weird.
I suppose we can merge it in any case if we find some regression (because of
the lack of distributed tests we could easily rollback into the previous
approach). For ref the command line to run this:
```bash
./gradlew :sdks:java:testing:load-tests:run -Prunner=":runners:spark"
-PloadTest.mainClass="org.apache.beam.sdk.loadtests.GroupByKeyLoadTest"
-PloadTest.args="--fanout=1 --iterations=1 --streaming=false
--runner=SparkStructuredStreamingRunner
--sourceOptions={\"numRecords\":200000,\"keySizeBytes\":100000,\"valueSizeBytes\":100000}"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 415121)
Time Spent: 15h (was: 14h 50m)
> Improve performance of GBK
> --------------------------
>
> Key: BEAM-9436
> URL: https://issues.apache.org/jira/browse/BEAM-9436
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
> Labels: structured-streaming
> Time Spent: 15h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
