etseidl commented on PR #9653:
URL: https://github.com/apache/arrow-rs/pull/9653#issuecomment-4218125311
I've now run multiple passes of the arrow_writer bench on my workstation and
there appear to be no regressions due to this PR. And the speed ups are quite
impressive 😄
<details>
```
group levels
main
----- ------
----
bool/bloom_filter 1.00 12.9±0.09ms
19.4 MB/sec 1.00 12.8±0.12ms 19.5 MB/sec
bool/default 1.00 8.5±0.06ms
29.3 MB/sec 1.00 8.5±0.09ms 29.3 MB/sec
bool/parquet_2 1.01 11.2±0.18ms
22.2 MB/sec 1.00 11.1±0.17ms 22.5 MB/sec
bool/zstd 1.00 9.0±0.10ms
27.8 MB/sec 1.00 9.0±0.10ms 27.9 MB/sec
bool/zstd_parquet_2 1.01 11.5±0.08ms
21.7 MB/sec 1.00 11.4±0.10ms 21.9 MB/sec
bool_non_null/bloom_filter 1.02 8.6±0.04ms
14.6 MB/sec 1.00 8.4±0.03ms 14.8 MB/sec
bool_non_null/default 1.05 2.9±0.01ms
42.4 MB/sec 1.00 2.8±0.04ms 44.4 MB/sec
bool_non_null/parquet_2 1.02 6.2±0.04ms
20.1 MB/sec 1.00 6.1±0.03ms 20.6 MB/sec
bool_non_null/zstd 1.05 3.3±0.04ms
38.2 MB/sec 1.00 3.1±0.06ms 40.1 MB/sec
bool_non_null/zstd_parquet_2 1.02 6.5±0.06ms
19.1 MB/sec 1.00 6.4±0.04ms 19.5 MB/sec
float_with_nans/bloom_filter 1.00 81.2±0.69ms
172.4 MB/sec 1.08 87.7±0.42ms 159.7 MB/sec
float_with_nans/default 1.00 58.0±0.86ms
241.4 MB/sec 1.08 62.8±0.28ms 222.9 MB/sec
float_with_nans/parquet_2 1.00 71.6±1.10ms
195.6 MB/sec 1.07 76.9±0.49ms 182.2 MB/sec
float_with_nans/zstd 1.00 88.6±0.36ms
158.0 MB/sec 1.07 94.6±0.36ms 148.0 MB/sec
float_with_nans/zstd_parquet_2 1.00 101.4±0.80ms
138.1 MB/sec 1.06 107.9±0.96ms 129.7 MB/sec
list_primitive/bloom_filter 1.06 319.5±1.83ms
1707.2 MB/sec 1.00 302.6±2.73ms 1802.0 MB/sec
list_primitive/default 1.07 260.7±1.76ms
2.0 GB/sec 1.00 242.8±1.50ms 2.2 GB/sec
list_primitive/parquet_2 1.00 257.0±1.68ms
2.1 GB/sec 1.00 257.5±3.19ms 2.1 GB/sec
list_primitive/zstd 1.01 390.4±2.65ms
1397.1 MB/sec 1.00 388.3±3.31ms 1404.6 MB/sec
list_primitive/zstd_parquet_2 1.03 387.2±2.82ms
1408.4 MB/sec 1.00 374.4±4.46ms 1456.7 MB/sec
list_primitive_non_null/bloom_filter 1.00 354.2±6.61ms
1536.5 MB/sec 1.02 360.1±4.36ms 1511.5 MB/sec
list_primitive_non_null/default 1.00 262.5±7.11ms
2.0 GB/sec 1.01 265.3±5.08ms 2.0 GB/sec
list_primitive_non_null/parquet_2 1.00 264.3±4.69ms
2.0 GB/sec 1.07 283.5±7.82ms 1919.6 MB/sec
list_primitive_non_null/zstd 1.01 527.5±10.36ms
1031.7 MB/sec 1.00 520.9±19.26ms 1044.7 MB/sec
list_primitive_non_null/zstd_parquet_2 1.00 510.5±7.07ms
1066.1 MB/sec 1.00 509.9±13.27ms 1067.4 MB/sec
list_primitive_sparse_99pct_null/bloom_filter 1.00 9.2±0.06ms
4.0 GB/sec 3.15 29.0±0.24ms 1288.8 MB/sec
list_primitive_sparse_99pct_null/default 1.00 8.7±0.08ms
4.2 GB/sec 3.30 28.6±0.64ms 1304.7 MB/sec
list_primitive_sparse_99pct_null/parquet_2 1.00 8.7±0.07ms
4.2 GB/sec 3.28 28.5±0.40ms 1310.8 MB/sec
list_primitive_sparse_99pct_null/zstd 1.00 10.3±0.10ms
3.5 GB/sec 2.91 29.9±0.21ms 1248.5 MB/sec
list_primitive_sparse_99pct_null/zstd_parquet_2 1.00 8.8±0.10ms
4.1 GB/sec 3.22 28.4±0.25ms 1315.2 MB/sec
primitive/bloom_filter 1.00 128.9±0.80ms
348.2 MB/sec 1.02 132.0±1.05ms 339.9 MB/sec
primitive/default 1.00 84.9±1.59ms
528.8 MB/sec 1.02 86.7±0.67ms 517.5 MB/sec
primitive/parquet_2 1.00 94.6±1.36ms
474.4 MB/sec 1.02 96.9±0.76ms 463.2 MB/sec
primitive/zstd 1.00 104.0±0.78ms
431.6 MB/sec 1.03 107.2±1.27ms 418.5 MB/sec
primitive/zstd_parquet_2 1.00 117.0±1.62ms
383.4 MB/sec 1.03 120.0±0.74ms 373.9 MB/sec
primitive_all_null/bloom_filter 1.00 1058.5±6.49µs
41.4 GB/sec 18.25 19.3±0.10ms 2.3 GB/sec
primitive_all_null/default 1.00 198.3±1.38µs
221.0 GB/sec 92.92 18.4±0.06ms 2.4 GB/sec
primitive_all_null/parquet_2 1.00 200.9±1.97µs
218.2 GB/sec 91.94 18.5±0.09ms 2.4 GB/sec
primitive_all_null/zstd 1.00 341.9±1.60µs
128.2 GB/sec 54.27 18.6±0.07ms 2.4 GB/sec
primitive_all_null/zstd_parquet_2 1.00 317.2±1.37µs
138.2 GB/sec 58.48 18.5±0.08ms 2.4 GB/sec
primitive_non_null/bloom_filter 1.00 94.8±1.16ms
464.1 MB/sec 1.10 103.9±0.44ms 423.5 MB/sec
primitive_non_null/default 1.00 38.5±0.22ms
1141.6 MB/sec 1.16 44.8±0.22ms 982.8 MB/sec
primitive_non_null/parquet_2 1.00 52.7±0.51ms
834.4 MB/sec 1.13 59.4±1.01ms 740.3 MB/sec
primitive_non_null/zstd 1.00 59.2±0.37ms
743.6 MB/sec 1.13 66.8±0.62ms 658.6 MB/sec
primitive_non_null/zstd_parquet_2 1.00 76.0±0.98ms
579.1 MB/sec 1.11 84.1±1.49ms 523.2 MB/sec
primitive_sparse_99pct_null/bloom_filter 1.00 12.9±0.27ms
3.4 GB/sec 2.23 28.8±0.70ms 1557.2 MB/sec
primitive_sparse_99pct_null/default 1.00 11.3±1.85ms
3.9 GB/sec 2.35 26.6±0.32ms 1686.3 MB/sec
primitive_sparse_99pct_null/parquet_2 1.00 11.6±1.71ms
3.8 GB/sec 2.30 26.8±0.28ms 1672.7 MB/sec
primitive_sparse_99pct_null/zstd 1.00 13.8±0.14ms
3.2 GB/sec 2.13 29.4±0.29ms 1528.3 MB/sec
primitive_sparse_99pct_null/zstd_parquet_2 1.00 12.4±0.06ms
3.5 GB/sec 2.27 28.1±0.28ms 1595.2 MB/sec
string/bloom_filter 1.00 169.3±11.30ms
3.0 GB/sec 1.05 178.1±13.47ms 2.9 GB/sec
string/default 1.05 121.8±12.92ms
4.2 GB/sec 1.00 116.3±3.32ms 4.4 GB/sec
string/parquet_2 1.03 120.8±6.66ms
4.2 GB/sec 1.00 117.6±1.10ms 4.4 GB/sec
string/zstd 1.00 308.2±4.22ms
1701.1 MB/sec 1.03 317.4±13.62ms 1651.6 MB/sec
string/zstd_parquet_2 1.01 287.9±2.18ms
1821.2 MB/sec 1.00 284.1±1.61ms 1845.6 MB/sec
string_and_binary_view/bloom_filter 1.00 48.8±0.29ms
661.4 MB/sec 1.01 49.3±0.35ms 654.5 MB/sec
string_and_binary_view/default 1.00 34.6±0.27ms
932.2 MB/sec 1.00 34.5±0.32ms 934.9 MB/sec
string_and_binary_view/parquet_2 1.01 43.9±0.28ms
734.1 MB/sec 1.00 43.7±0.31ms 738.4 MB/sec
string_and_binary_view/zstd 1.00 61.1±0.34ms
528.0 MB/sec 1.00 61.3±1.04ms 526.1 MB/sec
string_and_binary_view/zstd_parquet_2 1.00 53.6±0.63ms
601.6 MB/sec 1.00 53.6±0.58ms 602.2 MB/sec
string_dictionary/bloom_filter 1.00 76.4±0.63ms
3.4 GB/sec 1.42 108.4±0.44ms 2.4 GB/sec
string_dictionary/default 1.00 51.7±0.24ms
5.0 GB/sec 1.58 81.8±0.34ms 3.2 GB/sec
string_dictionary/parquet_2 1.00 55.5±0.66ms
4.6 GB/sec 1.50 83.5±0.55ms 3.1 GB/sec
string_dictionary/zstd 1.00 150.0±1.17ms
1760.5 MB/sec 1.08 162.3±7.72ms 1627.8 MB/sec
string_dictionary/zstd_parquet_2 1.00 142.7±0.88ms
1850.4 MB/sec 1.00 142.7±1.09ms 1850.5 MB/sec
string_non_null/bloom_filter 1.00 191.4±1.91ms
2.7 GB/sec 1.09 208.4±8.39ms 2.5 GB/sec
string_non_null/default 1.00 126.2±1.83ms
4.1 GB/sec 1.13 142.0±7.93ms 3.6 GB/sec
string_non_null/parquet_2 1.00 137.1±2.30ms
3.7 GB/sec 1.00 137.7±1.85ms 3.7 GB/sec
string_non_null/zstd 1.00 378.5±1.99ms
1384.4 MB/sec 1.06 400.3±7.49ms 1309.0 MB/sec
string_non_null/zstd_parquet_2 1.00 359.4±2.26ms
1458.0 MB/sec 1.04 372.0±7.03ms 1408.5 MB/sec
struct_all_null/bloom_filter 1.00 452.8±3.14µs
34.8 GB/sec 17.39 7.9±0.04ms 2047.7 MB/sec
struct_all_null/default 1.00 85.5±0.63µs
184.1 GB/sec 87.80 7.5±0.04ms 2.1 GB/sec
struct_all_null/parquet_2 1.00 86.5±1.38µs
182.0 GB/sec 86.71 7.5±0.03ms 2.1 GB/sec
struct_all_null/zstd 1.00 146.8±1.12µs
107.3 GB/sec 51.77 7.6±0.09ms 2.1 GB/sec
struct_all_null/zstd_parquet_2 1.00 136.4±1.14µs
115.4 GB/sec 55.50 7.6±0.06ms 2.1 GB/sec
struct_non_null/bloom_filter 1.00 41.0±0.59ms
390.6 MB/sec 1.29 53.0±0.27ms 301.8 MB/sec
struct_non_null/default 1.00 17.7±0.12ms
901.8 MB/sec 1.59 28.2±0.16ms 567.4 MB/sec
struct_non_null/parquet_2 1.00 23.3±0.13ms
686.6 MB/sec 1.46 34.1±0.20ms 469.3 MB/sec
struct_non_null/zstd 1.00 24.3±0.13ms
658.0 MB/sec 1.44 35.1±0.22ms 455.8 MB/sec
struct_non_null/zstd_parquet_2 1.00 33.6±0.19ms
476.6 MB/sec 1.31 44.1±0.46ms 363.0 MB/sec
struct_sparse_99pct_null/bloom_filter 1.00 5.9±0.04ms
2.7 GB/sec 2.11 12.4±0.15ms 1303.5 MB/sec
struct_sparse_99pct_null/default 1.00 5.0±0.04ms
3.2 GB/sec 2.32 11.6±0.11ms 1393.1 MB/sec
struct_sparse_99pct_null/parquet_2 1.00 5.0±0.03ms
3.2 GB/sec 2.32 11.6±0.13ms 1393.7 MB/sec
struct_sparse_99pct_null/zstd 1.00 6.2±0.04ms
2.6 GB/sec 2.07 12.8±0.19ms 1264.3 MB/sec
struct_sparse_99pct_null/zstd_parquet_2 1.00 5.6±0.03ms
2.8 GB/sec 2.16 12.1±0.13ms 1330.1 MB/sec
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
