yihua opened a new pull request, #826: URL: https://github.com/apache/hudi-rs/pull/826
## Description Adds a DuckDB leg to the TPC-H harness, so DuckDB's [hudi extension](https://github.com/alitrack/duckdb-hudi) (built on hudi-rs) can be timed and checked the same way as the DataFusion and Spark legs. `run.sh bench-duckdb` times `read_hudi` and `read_parquet` with the same query text, and `run.sh validate-duckdb` compares the two row by row. `make bench-tpch ENGINE=duckdb` calls the first. Each query runs in its own process, so a query that crashes DuckDB is recorded as failed and the run goes on. `--extension <path>` with `--label` times a locally built extension against the published one, and `--threads` caps DuckDB's cores. Running it found that the published extension (v0.2.1) returns wrong results or crashes on 17 of the 22 queries, fixed in alitrack/duckdb-hudi#1. A streaming scan with projection pushdown on top of that, alitrack/duckdb-hudi#2, brings SF1 from 65.4 s to 4.2 s, against 1.31 s for `read_parquet`. ## How are the changes test-covered - [ ] N/A - [ ] Automated tests (unit and/or integration tests) - [x] Manual tests - [x] Details are described below On DuckDB 1.5.6, at SF0.01 and SF1: `validate-duckdb` passes all 22 queries on a fixed extension build and reports the community build's crashes and differences per query. `bench-duckdb` runs with the community and local builds side by side, writing labelled result files that `compare --runs` reads. Also checked: `make bench-tpch ENGINE=duckdb`, rejection of a bad `--label` and of non-S3 cloud URLs, and a broken extension path reported once rather than once per query. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
