SteNicholas commented on code in PR #1308:
URL: https://github.com/apache/paimon-rust/pull/1308#discussion_r4439787781
##########
crates/integrations/datafusion/src/sql_context.rs:
##########
@@ -157,6 +158,7 @@ impl SQLContextBuilder {
let mut state_builder = SessionStateBuilder::new()
.with_config(crate::lateral_vector_search::session_config())
.with_default_features()
+ .with_spark_features()
Review Comment:
**[P2] Preserve DATE_PART result types when enabling the Spark planner**
Spark 54's `SparkDatePart::return_field_from_args` declares `Int32` for
every part, but its simplifier delegates `epoch` to native `date_part`, which
returns `Float64` (`nanosecond` similarly returns `Int64`). With this
registration, type coercion runs against the incorrect integer declaration
before simplification. For example:
```sql
SELECT EXTRACT(EPOCH FROM TIMESTAMP '1970-01-01 00:00:01.500') / 2;
```
Tracing the coercion and simplification paths gives integer `0` instead of
the previous `0.75`: the analyzer inserts an `Int64` cast around the extraction
and selects integer division, then simplification replaces the extraction with
a `Float64` expression inside that cast.
Please preserve the actual result types for these parts and add arithmetic
regression coverage for both `date_part` calls and EXTRACT syntax.
Re-registering `date_part` alone will not fix EXTRACT because
`SparkFunctionPlanner::plan_extract` constructs the upstream UDF directly.
Sources: [Spark
DATE_PART](https://github.com/apache/datafusion/blob/54.0.0/datafusion/spark/src/function/datetime/date_part.rs),
[native
DATE_PART](https://github.com/apache/datafusion/blob/54.0.0/datafusion/functions/src/datetime/date_part.rs),
[Spark
planner](https://github.com/apache/datafusion/blob/54.0.0/datafusion/spark/src/planner.rs).
This finding is based on source tracing; the SQL reproduction was not
executed locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]