Re: [I] Hash aggregation produces batches reporting huge memory size [datafusion]

via GitHub Thu, 28 May 2026 13:48:59 -0700


ariel-miculas commented on issue #22526:
URL: https://github.com/apache/datafusion/issues/22526#issuecomment-4568108875


   Another proof of this issue:
   ```
   $ cargo run --profile=release-nonlto --bin dfbench -- clickbench \
           --path benchmarks/data/hits_partitioned \
           --queries-path benchmarks/queries/clickbench/queries \
           --query 34 \
           -i 1 \
           --debug \
           --memory-limit 100G 2>&1
   
   Running benchmarks with the following options: RunOpt { query: Some(34), 
pushdown: false, common: CommonOpt { iterations: 1, partitions: None, 
batch_size: None, mem_pool_type: "fair", memory_limit: Some(107374182400), 
sort_spill_reservation_bytes: None, debug: true, simulate_latency: false }, 
path: "benchmarks/data/hits_partitioned", queries_path: 
"benchmarks/queries/clickbench/queries", output_path: None, sorted_by: None, 
sort_order: "ASC", config_options: [] }
   Q34: -- Must set for ClickBench hits_partitioned dataset. See 
https://github.com/apache/datafusion/issues/16591
   -- set datafusion.execution.parquet.binary_as_string = true
   
   SELECT 1, "URL", COUNT(*) AS c FROM hits GROUP BY 1, "URL" ORDER BY c DESC 
LIMIT 10;
   
   Query 34 iteration 0 took 1338.9 ms and returned 10 rows
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | plan_type         | plan                                                   
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                         |
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | Plan with Metrics | SortPreservingMergeExec: [c@2 DESC], fetch=10, 
metrics=[output_rows=10, elapsed_compute=3.12µs, output_bytes=18.2 MB, 
output_batches=1]                                                               
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   |                   |   SortExec: TopK(fetch=10), expr=[c@2 DESC], 
preserve_partitioning=[true], filter=[c@2 IS NULL OR c@2 > 55255], 
metrics=[output_rows=114, elapsed_compute=6.38ms, output_bytes=129.9 MB, 
output_batches=14, row_replacements=234]                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                    
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                         |
   |                   |     ProjectionExec: expr=[1 as Int64(1), URL@0 as URL, 
count(Int64(1))@1 as c], metrics=[output_rows=18.34 M, elapsed_compute=2.84ms, 
output_bytes=612.5 GB, output_batches=2.24 K, expr_0_eval_time=1.82ms, 
expr_1_eval_time=280.68µs, expr_2_eval_time=156.67µs]                           
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                              
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                           |
   |                   |       AggregateExec: mode=FinalPartitioned, gby=[URL@0 
as URL], aggr=[count(Int64(1))], metrics=[output_rows=18.34 M, 
elapsed_compute=2.70s, output_bytes=612.3 GB, output_batches=2.24 K, 
spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, peak_mem_used=9.47 B, 
aggregate_arguments_time=1.01ms, aggregation_time=78.08ms, 
emitting_time=70.33µs, time_calculating_group_ids=2.61s]                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   |                   |         RepartitionExec: partitioning=Hash([URL@0], 
14), input_partitions=14, metrics=[output_rows=22.83 M, elapsed_compute=2.09ms, 
output_bytes=5.6 GB, output_batches=2.79 K, spill_count=0, spilled_bytes=0.0 B, 
spilled_rows=0, fetch_time=11.11s, repartition_time=311.20ms, send_time=2.52s]  
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                         
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                         |
   |                   |           AggregateExec: mode=Partial, gby=[URL@0 as 
URL], aggr=[count(Int64(1))], metrics=[output_rows=22.83 M, 
elapsed_compute=6.08s, output_bytes=821.3 GB, output_batches=2.79 K, 
spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, skipped_aggregation_rows=0, 
peak_mem_used=5.44 B, aggregate_arguments_time=21.59ms, 
aggregation_time=193.54ms, emitting_time=84.50µs, 
time_calculating_group_ids=5.84s, reduction_factor=22.83% (22.83 M/100.00 M)]   
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   |                   |             DataSourceExec: file_groups={14 groups: 
[[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_0.parquet:0..122446530,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_1.parquet:0..174965044,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_10.parquet:0..101513258,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_11.parquet:0..118419888,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_12.parquet:0..149514164,
 ...], 
[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_15.parquet:88577877..103098894,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_16.parquet:0..101067219,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_17.parquet:0..116867853,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_18.parquet:0..133119589,
 Users/amiculas/work/datafusion/benchmarks/data/hits_partition
 ed/hits_19.parquet:0..103692598, ...], 
[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_23.parquet:73829085..79631107,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_24.parquet:0..78257049,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_25.parquet:0..144169728,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_26.parquet:0..156510916,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_27.parquet:0..166286210,
 ...], 
[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_30.parquet:67171810..124187913,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_31.parquet:0..123065410,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_32.parquet:0..94506004,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_33.parquet:0..78243765,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_34.parquet:0..119426616,
 
 ...], 
[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_39.parquet:94059938..103522954,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_4.parquet:0..140929275,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_40.parquet:0..142508647,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_41.parquet:0..290614269,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_42.parquet:0..288524057,
 ...], ...]}, projection=[URL], file_type=parquet, metrics=[output_rows=100.00 
M, elapsed_compute=4.45ms, output_bytes=19.9 GB, output_batches=12.31 K, 
files_ranges_pruned_statistics=113 total → 113 matched, 
row_groups_pruned_statistics=325 total → 325 matched, 
row_groups_pruned_bloom_filter=325 total → 325 matched, 
page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 
matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, 
bytes_scanned=2.62 B, file_open_er
 rors=0, file_scan_errors=0, files_opened=113, files_processed=113, 
num_predicate_creation_errors=0, predicate_evaluation_errors=0, 
pushdown_rows_matched=0, pushdown_rows_pruned=0, 
predicate_cache_inner_records=0, predicate_cache_records=0, 
bloom_filter_eval_time=3.90µs, metadata_load_time=2.11ms, 
page_index_eval_time=226ns, row_pushdown_eval_time=226ns, 
statistics_eval_time=226ns, time_elapsed_opening=8.92ms, 
time_elapsed_processing=4.33s, time_elapsed_scanning_total=11.09s, 
time_elapsed_scanning_until_data=313.52ms, output_rows_skew=2.18%, 
scan_efficiency_ratio=1637.04% (2.62 B/160.3 M)] |
   |                   |                                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                         |
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   Query 34 avg time: 1338.93 ms
   ```
   
   ```
   ❮ cargo run --profile=release-nonlto --bin dfbench -- clickbench \
           --path benchmarks/data/hits_partitioned \
           --queries-path benchmarks/queries/clickbench/queries \
           --query 34 \
           -i 1 \
           --debug \
           --memory-limit 600M 2>&1
   
       Finished `release-nonlto` profile [optimized] target(s) in 0.18s
        Running `target/release-nonlto/dfbench clickbench --path 
benchmarks/data/hits_partitioned --queries-path 
benchmarks/queries/clickbench/queries --query 34 -i 1 --debug --memory-limit 
600M`
   Running benchmarks with the following options: RunOpt { query: Some(34), 
pushdown: false, common: CommonOpt { iterations: 1, partitions: None, 
batch_size: None, mem_pool_type: "fair", memory_limit: Some(629145600), 
sort_spill_reservation_bytes: None, debug: true, simulate_latency: false }, 
path: "benchmarks/data/hits_partitioned", queries_path: 
"benchmarks/queries/clickbench/queries", output_path: None, sorted_by: None, 
sort_order: "ASC", config_options: [] }
   Q34: -- Must set for ClickBench hits_partitioned dataset. See 
https://github.com/apache/datafusion/issues/16591
   -- set datafusion.execution.parquet.binary_as_string = true
   
   SELECT 1, "URL", COUNT(*) AS c FROM hits GROUP BY 1, "URL" ORDER BY c DESC 
LIMIT 10;
   
   Query 34 iteration 0 took 7140.9 ms and returned 10 rows
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | plan_type         | plan                                                   
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | Plan with Metrics | SortPreservingMergeExec: [c@2 DESC], fetch=10, 
metrics=[output_rows=10, elapsed_compute=5.21µs, output_bytes=9.4 MB, 
output_batches=1]                                                               
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                           |
   |                   |   SortExec: TopK(fetch=10), expr=[c@2 DESC], 
preserve_partitioning=[true], filter=[c@2 IS NULL OR c@2 > 55255], 
metrics=[output_rows=90, elapsed_compute=7.06ms, output_bytes=68.1 MB, 
output_batches=14, row_replacements=241]                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   |                   |     ProjectionExec: expr=[1 as Int64(1), URL@0 as URL, 
count(Int64(1))@1 as c], metrics=[output_rows=18.34 M, elapsed_compute=3.06ms, 
output_bytes=5.1 GB, output_batches=3.09 K, expr_0_eval_time=2.02ms, 
expr_1_eval_time=195.81µs, expr_2_eval_time=82.64µs]                            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                            |
   |                   |       AggregateExec: mode=FinalPartitioned, gby=[URL@0 
as URL], aggr=[count(Int64(1))], metrics=[output_rows=43.53 M, 
elapsed_compute=13.57s, output_bytes=19.1 GB, output_batches=6.17 K, 
spill_count=1.11 K, spilled_bytes=17.3 GB, spilled_rows=103.2 M, 
peak_mem_used=260.8 M, aggregate_arguments_time=2.26ms, 
aggregation_time=60.97ms, emitting_time=4.52ms, 
time_calculating_group_ids=2.49s]                                               
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                         
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   |                   |         RepartitionExec: partitioning=Hash([URL@0], 
14), input_partitions=14, metrics=[output_rows=25.99 M, elapsed_compute=6.15ms, 
output_bytes=5.9 GB, output_batches=3.18 K, spill_count=10, spilled_bytes=45.2 
MB, spilled_rows=237.6 K, fetch_time=9.46s, repartition_time=381.07ms, 
send_time=1.79s]                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                   
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   |                   |           AggregateExec: mode=Partial, gby=[URL@0 as 
URL], aggr=[count(Int64(1))], metrics=[output_rows=25.99 M, 
elapsed_compute=4.32s, output_bytes=42.9 GB, output_batches=3.18 K, 
spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, skipped_aggregation_rows=0, 
peak_mem_used=230.1 M, aggregate_arguments_time=23.73ms, 
aggregation_time=153.48ms, emitting_time=131.38ms, 
time_calculating_group_ids=3.95s, reduction_factor=25.99% (25.99 M/100.00 M)]   
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   |                   |             DataSourceExec: file_groups={14 groups: 
[[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_0.parquet:0..122446530,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_1.parquet:0..174965044,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_10.parquet:0..101513258,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_11.parquet:0..118419888,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_12.parquet:0..149514164,
 ...], 
[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_15.parquet:88577877..103098894,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_16.parquet:0..101067219,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_17.parquet:0..116867853,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_18.parquet:0..133119589,
 Users/amiculas/work/datafusion/benchmarks/data/hits_partition
 ed/hits_19.parquet:0..103692598, ...], 
[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_23.parquet:73829085..79631107,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_24.parquet:0..78257049,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_25.parquet:0..144169728,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_26.parquet:0..156510916,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_27.parquet:0..166286210,
 ...], 
[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_30.parquet:67171810..124187913,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_31.parquet:0..123065410,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_32.parquet:0..94506004,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_33.parquet:0..78243765,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_34.parquet:0..119426616,
 
 ...], 
[Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_39.parquet:94059938..103522954,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_4.parquet:0..140929275,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_40.parquet:0..142508647,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_41.parquet:0..290614269,
 
Users/amiculas/work/datafusion/benchmarks/data/hits_partitioned/hits_42.parquet:0..288524057,
 ...], ...]}, projection=[URL], file_type=parquet, metrics=[output_rows=100.00 
M, elapsed_compute=3.95ms, output_bytes=19.9 GB, output_batches=12.31 K, 
files_ranges_pruned_statistics=113 total → 113 matched, 
row_groups_pruned_statistics=325 total → 325 matched, 
row_groups_pruned_bloom_filter=325 total → 325 matched, 
page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 
matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, 
bytes_scanned=2.62 B, file_open_er
 rors=0, file_scan_errors=0, files_opened=113, files_processed=113, 
num_predicate_creation_errors=0, predicate_evaluation_errors=0, 
pushdown_rows_matched=0, pushdown_rows_pruned=0, 
predicate_cache_inner_records=0, predicate_cache_records=0, 
bloom_filter_eval_time=5.47µs, metadata_load_time=2.15ms, 
page_index_eval_time=226ns, row_pushdown_eval_time=226ns, 
statistics_eval_time=226ns, time_elapsed_opening=10.48ms, 
time_elapsed_processing=4.12s, time_elapsed_scanning_total=17.84s, 
time_elapsed_scanning_until_data=348.59ms, output_rows_skew=3.54%, 
scan_efficiency_ratio=1637.04% (2.62 B/160.3 M)] |
   |                   |                                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                          |
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   Query 34 avg time: 7140.90 ms
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [I] Hash aggregation produces batches reporting huge memory size [datafusion]

Reply via email to