Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21733
@tdas
Done running perf. test with 4 more tests:
> BenchmarkMovingAggregationsListenerKeyMuchBigger
rate: 160000
version | input rows per second | processed rows per second | total state
rows | used bytes of current state version
---- | ---- | ---- | ---- | ----
| latest master (c9914cf) | 159877.232 | 149537.817 | 65000 | 133511303 |
| patch (on top of c9914cf) | 160049.118 | 152497.945 | 65000 | 73236351 |
state size: 54.854 % (reduces 45.15%)
> BenchmarkMovingAggregationsListenerManyKeys
rate: 120000
version | input rows per second | processed rows per second | total state
rows | used bytes of current state version
---- | ---- | ---- | ---- | ----
| latest master (c9914cf) | 120266.810 | 107482.042 | 65000 | 38433719 |
| patch (on top of c9914cf) | 119865.855 | 109268.772 | 65000 | 24900343 |
state size: 64.787% (reduces 35.21%)
> BenchmarkMovingAggregationsListenerManyValues
rate: 25000
version | input rows per second | processed rows per second | total state
rows | used bytes of current state version
---- | ---- | ---- | ---- | ----
| latest master (c9914cf) | 25009.236 | 21216.126 | 90000 | 77161711
(857.352 per row) |
| patch (on top of c9914cf) | 25060.635 | 20774.500 | 99495 | 78230335
(786.274 per row) |
state size: 91.709 % (reduces 8.29 %)
> BenchmarkMovingAggregationsListenerValueMuchBigger
rate: 85000
version | input rows per second | processed rows per second | total state
rows | used bytes of current state version
---- | ---- | ---- | ---- | ----
| latest master (c9914cf) | 85310.774 | 79091.271 | 1000 | 1324255 |
| patch (on top of c9914cf) | 84791.761 | 79755.905 | 1000 | 1282687 |
state size: 96.861 % (reduces 3.14 %)
I don't find any outstanding perf. hit, and expected state size reduction
is shown from all over the cases.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
