andygrove opened a new pull request, #27165: URL: https://github.com/apache/datafusion/pull/27165
## Which issue does this PR close? - Part of #17045. ## Rationale for this change The expected results in `datafusion/sqllogictest/test_files/spark` come from DataFusion, and nothing checks them against Spark (#27103). Running some of these files on a local Spark has already found wrong expectations (#26333, #26338), but everyone who does that has to build their own harness. This adds one as a manual tool, so anyone with Python and Java can check one file, or the whole directory, against Spark 4.2. It is a smaller version of #21508, which was closed. That PR ran in CI on three Spark versions and translated DataFusion SQL to Spark SQL with regular expressions. This one runs only by hand, only on Spark 4.2, and skips what Spark cannot parse instead of translating it. Spark 4.2 parses `::` casts itself, which avoids a precedence bug in the old translator: it rewrote `-128::TINYINT` as `CAST(-128 AS TINYINT)`, but DataFusion and Spark both parse it as `-(128::TINYINT)`. ## What changes are included in this PR? - `datafusion/spark/scripts/pyspark_slt_runner.py` runs each record on a local PySpark 4.2 session. It reports the records where Spark returns different rows, raises an error the file does not expect (or does not raise one the file expects), returns a different column type letter, or returns the same rows in a different order. It prints Spark's values the way the sqllogictest engine prints DataFusion's (15 decimal places for doubles in `spark/` files, decimals at their full scale, `(empty)` for an empty string), so the comparison is plain text. - The runner passes `enable_ansi_mode`, `time_zone` and `map_key_dedup_policy` on to Spark, turns `CREATE TABLE` into temporary views, and renames PostgreSQL type names (`FLOAT8`, `TEXT`, ...) and `columnN`. It skips queries Spark cannot parse, calls to functions Spark does not have (`arrow_cast`, `arrow_typeof`), string literals that contain a backslash (Spark reads it as an escape and DataFusion's Spark dialect does not), and decimals with a precision above 38. - A README with setup and usage, linked from the Spark sqllogictest README. - `__pycache__/` in `.gitignore`, since running the unit tests creates one next to the script. Nothing runs in CI. ## What is the testing strategy for this PR? `test_pyspark_slt_runner.py` has unit tests for the parsing, the translation, the value formatting (against the cases in `conversion.rs`) and the classification of each record, using a fake Spark session. They need neither Spark nor Java: ```shell python3 -m unittest discover -s datafusion/spark/scripts ``` The parser and translator also run without errors over all 245 files (2,715 records). This draft has not been run against Spark yet. I will add the results of a full run before marking it ready for review. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
