blake-hu opened a new pull request, #1990: URL: https://github.com/apache/iggy/pull/1990
## Description Currently, the build() method from BenchmarkReportBuilder is single-threaded. This PR parallelizes this in 3 places: 1. In build(), spawn a new thread for every call to `from_individual_metrics` or `from_producers_and_consumers_statistics`. 2. In calculating aggregated time series, spawn one thread for each time series (MB, msg, latency). 3. In TimeSeriesCalculator, use rayon parallel iterators. (I also tried implementing multithreading and `par_iter`s in other places, but those led to slower performance, likely due to threading overhead. Those are not included in the PR.) ## Measuring speedup I measured the average runtime of build() with 200 producers and 200 consumers on a Ryzen 9 7950X with 124 GB RAM. build() shows between 3.7-11x speedup, with greater speedup for greater message loads. | Message count | Single-threaded runtime (ms) | Multi-threaded runtime (ms) | Speedup | | ------------- | ---------------------------- | --------------------------- | ------- | | 8M messages | 155 | 42 | 3.7x | | 16M messages | 497 | 74 | 6.7x | | 32M messages | 1791 | 162 | 11.0x | Closes #1976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
