AnuragRaut08 opened a new pull request, #27425: URL: https://github.com/apache/datafusion/pull/27425
## Which issue does this PR close? - Closes #27406. ## Rationale for this change The TPC-H benchmark data generation commands in `benchmarks/bench.sh` are incompatible with `tpchgen-cli` v4. Version 4 introduced breaking CLI changes, removing the legacy `--format` and `--parquet-compression` options in favor of format-specific subcommands and the `--compression` option. As a result, users running the benchmark data generation commands with `tpchgen-cli` v4 encounter errors and cannot generate the required TPC-H datasets. This PR updates the benchmark script to restore compatibility with `tpchgen-cli` v4. ## What changes are included in this PR? Updated `benchmarks/bench.sh` to use the `tpchgen-cli` v4 syntax: - **Parquet generation:** Replaced `--format parquet` with the `parquet` subcommand and `--parquet-compression` with `--compression`. - **CSV generation:** Replaced `--format csv` with the `csv` subcommand. - **Sort-pushdown generation:** Updated the Parquet generation command in `data_sort_pushdown()` to use the new CLI syntax. - **Installation hint:** Updated the suggested installation command to install a compatible v4 release using `cargo install tpchgen-cli --version ^4 --locked`. Existing scale factors, compression settings, partition counts, output directories, and sort-pushdown file-renaming behavior are preserved. ## What is the testing strategy for this PR? Validated the updated commands using `tpchgen-cli 4.0.0` by executing the following benchmark data-generation targets: - `./bench.sh data tpch` - `./bench.sh data tpch10` - `./bench.sh data tpch_csv` - `./bench.sh data tpch_csv10` - `./bench.sh data sort_pushdown` All five targets completed successfully. Additionally, verified that: - Parquet datasets were generated successfully at scale factors 1 and 10. - CSV datasets were generated successfully at scale factors 1 and 10, with the expected `<table>/<table>.1.csv` output layout. - Sort-pushdown generated the expected three Parquet files: `a_part3.parquet`, `b_part2.parquet`, and `c_part1.parquet`. - `bash -n benchmarks/bench.sh` passes. - `git diff --check` passes. No new automated tests were added because this change updates external CLI invocation syntax in the benchmark script. The affected commands were exercised directly against `tpchgen-cli 4.0.0`. ## Are there any user-facing changes? Yes. Users can now generate TPC-H Parquet, CSV, and sort-pushdown benchmark datasets using `tpchgen-cli` v4 without encountering errors from the removed legacy CLI options. The installation hint also provides the correct command for installing a compatible v4 release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
