andygrove opened a new pull request, #368:
URL: https://github.com/apache/datafusion-spark/pull/368

   ## Which issue does this PR close?
   
   - Relates to #7.
   
   ## Rationale for this change
   
   The PySpark runner reports 260 records in the `.slt` files whose expected 
result differs from Spark 4.2. Most are known bugs with an issue, some come 
from how the test is written, and some are limitations of the runner. They are 
reported together with any new difference, so a new difference is easy to miss 
and the runner always exits with status 1.
   
   ## What changes are included in this PR?
   
   - A comment directly above a record marks a known difference: `# xfail 
spark: <reason>`. It is a comment because the Rust sqllogictest parser rejects 
directives it does not know.
   - A marked record that differs from Spark counts as an expected failure. It 
is not reported and does not change the exit status. `--verbose` lists it with 
its reason.
   - A marked record that passes is reported as `marked xfail, but Spark 
agrees`, so the comment is removed once the difference is fixed. A marked 
record that the runner skips stays skipped.
   - The 260 records that differ today are marked. Every bug and literal-typing 
reason links an issue. The other reasons say whether the difference comes from 
the test, is intended, or is a runner limitation, and why.
   
   | Cause | Records |
   | --- | --- |
   | Bug, with a linked issue | 148 |
   | Literal typing, #97 | 63 |
   | Intended difference, such as `json_tuple` and `stack` returning a struct 
or list instead of rows | 29 |
   | Runner limitation | 14 |
   | Test written in a way Spark reads differently | 6 |
   
   Ten of the bug records had no issue, so this PR links new ones: #361, #362, 
#363, #364, #365, #366 and #367.
   
   The runner-limitation records come from translation gaps: 10 are a setting 
the runner does not pass on to Spark (`spark.map_key_dedup_policy`), 3 are 
DataFusion's `TIMESTAMP` literal, which is `TIMESTAMP_NTZ`, and 1 is `[]` 
indexing, which starts at 1 in DataFusion and at 0 in Spark. Those can be fixed 
in the runner in follow-up PRs, which removes their markers.
   
   ## Are these changes tested?
   
   Unit tests in `test_pyspark_slt_runner.py` cover parsing the comment, 
expected failures, a marked record that passes, and a marked record that is 
skipped.
   
   With PySpark 4.2.0, the full run now reports 2907 passing records, 260 
expected failures and no differences. `cargo test -p 
datafusion-spark-sqllogictest` still parses every file.
   
   ## Are there any user-facing changes?
   
   No. This changes only the test files and the PySpark runner.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to