Rich-T-kid commented on PR #26579: URL: https://github.com/apache/datafusion/pull/26579#issuecomment-6329707252
### Parquet scan time: dictionary vs. Utf8View Fixed fork `fb5bbbae0350faf5bb322e7d69e37e8ce75ed687`; ClickBench partitioned (100 files); 12 target partitions; 32 GB memory limit; Apple M4 Max (16 cores); optimized release build. Only `datafusion.execution.parquet.enable_rle_to_dictionary` changes. Both sides use `binary_as_string=true` and `schema_force_view_types=true`; RLE off produces `Utf8View`, while RLE on produces `Dictionary<Int32, Utf8>`. Values are the median of five `EXPLAIN ANALYZE` runs. `Parquet` is the aggregate `elapsed_compute` metric across scan partitions on the current `DataSourceExec` parquet node (formerly `ParquetExec`), so it is operator compute rather than wall-clock query time. Positive delta means dictionary decoding took longer. Q0 and Q6 are metadata-only and do not execute a scan node. Summary: the sum of per-query scan medians is 82.41s for Utf8View and 122.16s for dictionary (1.48x). Dictionary is more than 10% slower on 24 queries, more than 10% faster on 4, and within 10% on 13. The conversion-focused Q10 and Q11 scans are 9.5% and 6.1% faster with dictionary reads, respectively. | Query | Utf8View Parquet | Dictionary Parquet | Dict / View | |---:|---:|---:|---:| | Q0 | N/A | N/A | metadata-only | | Q1 | 29.84 ms | 29.54 ms | 0.99× (-1.0%) | | Q2 | 200.6 ms | 216.6 ms | 1.08× (+8.0%) | | Q3 | 163.2 ms | 166.2 ms | 1.02× (+1.8%) | | Q4 | 169.1 ms | 180.8 ms | 1.07× (+6.9%) | | Q5 | 921.3 ms | 1490.0 ms | 1.62× (+61.7%) | | Q6 | N/A | N/A | metadata-only | | Q7 | 36.84 ms | 34.64 ms | 0.94× (-6.0%) | | Q8 | 448.3 ms | 395.3 ms | 0.88× (-11.8%) | | Q9 | 594.5 ms | 612.7 ms | 1.03× (+3.1%) | | Q10 | 274.5 ms | 248.4 ms | 0.90× (-9.5%) | | Q11 | 340.6 ms | 319.7 ms | 0.94× (-6.1%) | | Q12 | 808.7 ms | 1460.0 ms | 1.81× (+80.5%) | | Q13 | 1100.0 ms | 1610.0 ms | 1.46× (+46.4%) | | Q14 | 1020.0 ms | 1580.0 ms | 1.55× (+54.9%) | | Q15 | 210.5 ms | 186.4 ms | 0.89× (-11.4%) | | Q16 | 1130.0 ms | 1580.0 ms | 1.40× (+39.8%) | | Q17 | 1120.0 ms | 1570.0 ms | 1.40× (+40.2%) | | Q18 | 1640.0 ms | 2030.0 ms | 1.24× (+23.8%) | | Q19 | 150.5 ms | 126.6 ms | 0.84× (-15.9%) | | Q20 | 4330.0 ms | 7040.0 ms | 1.63× (+62.6%) | | Q21 | 5030.0 ms | 8460.0 ms | 1.68× (+68.2%) | | Q22 | 9920.0 ms | 16250.0 ms | 1.64× (+63.8%) | | Q23 | 29440.0 ms | 39320.0 ms | 1.34× (+33.6%) | | Q24 | 240.9 ms | 368.3 ms | 1.53× (+52.9%) | | Q25 | 797.6 ms | 1380.0 ms | 1.73× (+73.0%) | | Q26 | 225.5 ms | 351.8 ms | 1.56× (+56.0%) | | Q27 | 4300.0 ms | 7770.0 ms | 1.81× (+80.7%) | | Q28 | 3780.0 ms | 7030.0 ms | 1.86× (+86.0%) | | Q29 | 202.2 ms | 184.5 ms | 0.91× (-8.7%) | | Q30 | 1340.0 ms | 1910.0 ms | 1.43× (+42.5%) | | Q31 | 1430.0 ms | 2170.0 ms | 1.52× (+51.7%) | | Q32 | 674.4 ms | 688.5 ms | 1.02× (+2.1%) | | Q33 | 4950.0 ms | 7280.0 ms | 1.47× (+47.1%) | | Q34 | 4860.0 ms | 7420.0 ms | 1.53× (+52.7%) | | Q35 | 199.0 ms | 165.0 ms | 0.83× (-17.1%) | | Q36 | 75.71 ms | 108.3 ms | 1.43× (+43.0%) | | Q37 | 31.95 ms | 52.81 ms | 1.65× (+65.3%) | | Q38 | 72.15 ms | 133.3 ms | 1.85× (+84.7%) | | Q39 | 132.9 ms | 217.1 ms | 1.63× (+63.3%) | | Q40 | 9.30 ms | 8.82 ms | 0.95× (-5.2%) | | Q41 | 8.57 ms | 8.46 ms | 0.99× (-1.3%) | | Q42 | 6.06 ms | 5.72 ms | 0.94× (-5.6%) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
