sovsparrow commented on issue #441: URL: https://github.com/apache/parquet-format/issues/441#issuecomment-5268862269
@alkis — tbh, your comment here inspired me to build [Parquity](https://github.com/sovsparrow/parquity). I gave you credit in the README as well. I went for a more local direction though. Parquity is now a Python CLI that tests the Parquet engines installed in the current environment. `check` runs a known table through a writer-by-reader matrix, `fuzz` generates tables and minimizes failing cases using Hypothesis, `scan` compares readers on files you already have, and `replay` re-runs saved reproducers. 0.2.0 supports PyArrow, DuckDB, Polars, DataFusion, and fastparquet. Running locally means there's not a shared CI job to keep every engine alive. However, it also means Parquity only reaches engines that can be installed in a Python environment. It doesn't solve the closed-source, GPU, or driver-distribution problem that has been discussed here. I already found two real bugs using it. `fuzz` hit a [DuckDB writer bug](https://github.com/duckdb/duckdb/issues/24731), and `scan` found a [Polars reader bug](https://github.com/pola-rs/polars/issues/28787) in Apache's `parquet-testing` corpus. If anyone wants to help take it further, including beyond the current Python-only boundary, contributions and ideas are totally welcome. Thanks for the inspiration 🤝 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
