andygrove opened a new pull request, #391:
URL: https://github.com/apache/datafusion-spark/pull/391

   ## Which issue does this PR close?
   
   - None. This is a follow-up to the PySpark runner CI job.
   
   ## Rationale for this change
   
   CI checks the `.slt` files against Spark 4.2 only. Users also run Spark 3.5, 
and it behaves differently in places. This PR adds the latest 3.5 release 
(3.5.9) to CI, so a new difference from 3.5 shows up in the PR that adds it.
   
   ## What changes are included in this PR?
   
   - **CI:** the `spark` job in `pyspark-runner.yml` is now a matrix over Spark 
3.5 (`requirements-spark-3.5.txt`, `pyspark==3.5.9`) and 4.2 
(`requirements.txt`), with `fail-fast: false`. The job is not a required check, 
so its new name does not affect `.asf.yaml`.
   - **Runner support for Spark 3.5:**
     - The runner accepts PySpark 3.5 and 4.2 (`SUPPORTED_VERSIONS`).
     - Spark 3.5 has no `::` syntax, so on Spark 3 the runner rewrites 
`expr::type` to `CAST(expr AS type)`. It follows Spark 4's precedence: `::` 
binds to the literal, column, call, subscript or parenthesized expression 
before it, and a minus sign belongs to a number literal (`-128::tinyint` is 
`CAST(-128 AS tinyint)`). Without this rewrite, 1488 records failed to parse on 
3.5. After it, 113 do, the same number as on 4.2.
     - PySpark 3.5 cannot read a schema that holds an interval 
(`CANNOT_PARSE_DATATYPE`). Those records are now skipped as a 
`CalendarIntervalType` value, as they already are on 4.2.
     - The runner works without `pyspark.logger`, which PySpark 3.5 does not 
have.
   - **Version conditions on xfail markers:** `# xfail spark<4.0: <reason>` 
applies only to the Spark versions that match. The operators are `<`, `<=`, 
`>`, `>=` and `==`, compared on major.minor, and conditions separated by commas 
must all hold (`spark>=4.0, <4.2`). A record can have several markers, and the 
first that matches applies. Plain `# xfail spark:` still applies to every 
version. The runner stops with an error if it cannot read a condition.
   - **Markers for the records where Spark 3.5 differs from Spark 4:**
     - `encode` (7 records): Spark 3.5 accepts any Java charset (`utf-32be`, 
`utf-32le`) and replaces unmappable characters with `?`. Spark 4 raises an 
error unless `spark.sql.legacy.javaCharsets` or 
`spark.sql.legacy.codingErrorAction` is set (checked on 4.2).
     - `greatest` (2 records): Spark 3.5 picks `DECIMAL(38, 10)` as the wider 
type of `DECIMAL(38, 10)` and `DECIMAL(38, 0)`, where Spark 4 picks 
`DECIMAL(38, 0)`.
     - `to_json` (1 record): Spark 3.5 drops the seconds of a local mean time 
offset (`-07:52` instead of `-07:52:58`).
     - `abs` (2 records): Spark 3.5 agrees with DataFusion on large `DECIMAL` 
values, so these markers become `spark>=4.0`.
   
   ## Are these changes tested?
   
   - New unit tests in `test_pyspark_slt_runner.py` cover the `::` rewrite, 
version conditions, stacked markers, invalid conditions and markers for other 
versions. The tests pass on Python 3.9 and 3.12.
   - Full local runs, both exiting with status 0:
     - Spark 4.2.0: 3093 pass, 270 expected failures, 606 skipped, unchanged 
from `main`.
     - Spark 3.5.9: 2792 pass, 277 expected failures, 900 skipped. Most of the 
extra skips call functions that Spark 3.5 does not have, such as 
`try_parse_url`, `make_time` and `is_valid_utf8`.
   - `cargo test -p datafusion-spark-sqllogictest` passes. The `.slt` changes 
are comments only.
   
   ## Are there any user-facing changes?
   
   No. The changes are to CI and the test runner only.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to