andygrove opened a new issue, #77: URL: https://github.com/apache/datafusion-spark/issues/77
### Describe the bug `date_part` differs from Spark in two ways: 1. It accepts field names that Spark rejects. `SparkDatePart::simplify` renames a few Spark aliases and passes every other name to DataFusion's `date_part` ([date_part.rs#L99-L104](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/spark/src/function/datetime/date_part.rs#L99-L104)), so any name that DataFusion's `date_part` knows is accepted. Spark raises `INVALID_EXTRACT_FIELD` for those names. 2. A NULL field is an internal error ([date_part.rs#L88-L96](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/spark/src/function/datetime/date_part.rs#L88-L96)). Spark returns NULL. ### To Reproduce Spark 4.2.0: ```sql SELECT date_part('QUARTERS', DATE '2000-01-01'); -- [INVALID_EXTRACT_FIELD] Cannot extract `QUARTERS` from "CAST(2000-01-01 AS DATE)". SQLSTATE: 42601 SELECT date_part('DAYOFYEAR', DATE '2000-01-01'); -- [INVALID_EXTRACT_FIELD] Cannot extract `DAYOFYEAR` from "CAST(2000-01-01 AS DATE)". SQLSTATE: 42601 SELECT date_part(NULL::string, DATE '2000-01-01'); -- NULL ``` DataFusion, existing tests: - [date_part.slt#L49](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/sqllogictest/test_files/datetime/date_part.slt#L49) and [#L135](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/sqllogictest/test_files/datetime/date_part.slt#L135) expect `1` for `'QUARTERS'` and `'DAYOFYEAR'`. - [date_part.slt#L223-L224](https://github.com/apache/datafusion-spark/blob/fc2babe8c153d95e70fd303d55368a800d90930c/crates/sqllogictest/test_files/datetime/date_part.slt#L223-L224) expects `Internal error: First argument of `DATE_PART` must be non-null scalar Utf8` for a NULL field. ### Expected behavior Accept only the field names Spark accepts, and raise an error like Spark's `INVALID_EXTRACT_FIELD` for the others. Return NULL for a NULL field. The expectations listed above need to change. ### Additional context Found by running the sqllogictest files on Spark 4.2.0 with the PySpark runner from #54 (ANSI mode off, UTC session). The Spark results above come from that run. DataFusion results are the expectations in the `.slt` files at fc2babe8, which CI enforces. #38 covers the fractional seconds that `date_part('SECOND', ...)` drops. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
