iemejia commented on issue #11055: [BEAM-9436] Improve GBK in spark structured streaming runner URL: https://github.com/apache/beam/pull/11055#issuecomment-608138463 Code looks ok. I could not consistently reproduce the results, when I run the load test in the branch vs master (with the gradleRun patch to include spark-sql) I see results that vary a lot, RAM compsumption is almost equal, but runtime tends to be faster but mostly by a tiny difference on this branch but some runs curiously take 4 more times than others (in both branches), so weird. I suppose we can merge it in any case if we find some regression (because of the lack of distributed tests we could easily rollback into the previous approach). For ref the command line to run this: ./gradlew :sdks:java:testing:load-tests:run -Prunner=":runners:spark" -PloadTest.mainClass="org.apache.beam.sdk.loadtests.GroupByKeyLoadTest" -PloadTest.args="--fanout=1 --iterations=1 --streaming=false --runner=SparkStructuredStreamingRunner --sourceOptions={\"numRecords\":200000,\"keySizeBytes\":100000,\"valueSizeBytes\":100000}"
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
