mingmwang commented on PR #6065:
URL:
https://github.com/apache/arrow-datafusion/pull/6065#issuecomment-1518931085
@comphead
Sorry that, I had tested this PR on my Mac locally and do not see any
performance improvement on tpch-q17(high cardinality aggregation), and there is
about 10% downgrade.
**Before this PR**:
q17(sf =1)
Running benchmarks with the following options: DataFusionBenchmarkOpt {
query: Some(17), debug: false, iterations: 3, partitions: 1, batch_size: 8192,
path: "./parquet_data", file_format: "parquet", mem_table: false, output_path:
None, disable_statistics: true, enable_scheduler: false }
Query 17 iteration 0 took 1942.3 ms and returned 1 rows
Query 17 iteration 1 took 1918.5 ms and returned 1 rows
Query 17 iteration 2 took 1940.6 ms and returned 1 rows
Query 17 avg time: 1933.79 ms
q17(sf =10)
Running benchmarks with the following options: DataFusionBenchmarkOpt {
query: Some(17), debug: false, iterations: 3, partitions: 1, batch_size: 8192,
path: "./parquet_data10", file_format: "parquet", mem_table: false,
output_path: None, disable_statistics: true, enable_scheduler: false }
Query 17 iteration 0 took 32352.1 ms and returned 1 rows
Query 17 iteration 1 took 32210.5 ms and returned 1 rows
Query 17 iteration 2 took 32003.7 ms and returned 1 rows
Query 17 avg time: 32188.74 ms
**this PR**:
q17(sf =1)
Running benchmarks with the following options: DataFusionBenchmarkOpt {
query: Some(17), debug: false, iterations: 3, partitions: 1, batch_size: 8192,
path: "./parquet_data", file_format: "parquet", mem_table: false, output_path:
None, disable_statistics: true, enable_scheduler: false }
Query 17 iteration 0 took 2159.3 ms and returned 1 rows
Query 17 iteration 1 took 2106.7 ms and returned 1 rows
Query 17 iteration 2 took 2111.4 ms and returned 1 rows
Query 17 avg time: 2125.82 ms
q17(sf =10)
Running benchmarks with the following options: DataFusionBenchmarkOpt {
query: Some(17), debug: false, iterations: 3, partitions: 1, batch_size: 8192,
path: "./parquet_data10", file_format: "parquet", mem_table: false,
output_path: None, disable_statistics: true, enable_scheduler: false }
Query 17 iteration 0 took 34505.3 ms and returned 1 rows
Query 17 iteration 1 took 34240.6 ms and returned 1 rows
Query 17 iteration 2 took 33967.1 ms and returned 1 rows
Query 17 avg time: 34237.68 ms
-------------------------------------
How this was tested:
```rust
if matches!(self.mode, AggregateMode::Partial | AggregateMode::Single)
&& normal_aggr_input_values.is_empty()
&& normal_filter_values.is_empty()
&& groups_with_rows.len() >= batch.num_rows() / 10
```
change the magic number from `10` to `1`
```rust
if matches!(self.mode, AggregateMode::Partial | AggregateMode::Single)
&& normal_aggr_input_values.is_empty()
&& normal_filter_values.is_empty()
&& groups_with_rows.len() >= batch.num_rows() / 1
```
So that the update accumulators will use the method
`update_accumulators_using_batch()` and call the method
`slice_and_maybe_filter()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
