Amar3tto commented on PR #26862: URL: https://github.com/apache/beam/pull/26862#issuecomment-1563291561
> @Amar3tto Thanks! > > Could you explain a bit why this change accelerate the test execution? > > Also, how did you calculate that now it takes only 9.53 min for this test? I see that [this build](https://ci-beam.apache.org/job/beam_PerformanceTests_SparkReceiver_IO/467/), that was run against this PR, took 36 mins. Do I misunderstand something? This change prevents splitting restriction too often (now it won't split until there are no more records left in the queue), which was the reason for the Receivers to start and stop too many times - starting and stopping takes a while. I ran the `SparkReceiverIO Performance test` 2 times on this PR. I got the results from the Grafana dashboard (9.53 and 18.3 min). Previous successful runs (before fix) have been around 27 min, too close to the 30 min timeout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
