andygrove opened a new pull request, #345:
URL: https://github.com/apache/datafusion-spark/pull/345

   ## Which issue does this PR close?
   
   - Relates to #332, #331, #330 and #70, which each carry their own copy of 
these rules. Requested in [this review 
comment](https://github.com/apache/datafusion-spark/pull/332#discussion_r4602650086).
   
   ## Rationale for this change
   
   Functions that need one type for several arguments (`array_position` now, 
`greatest`, `least`, `array_remove`, `array_max`, `array_min` and 
`array_insert` in open PRs) each carry a copy of Spark's common-type rules. The 
copies have already drifted apart. This PR gives them one shared copy, so fixes 
land once.
   
   It also fixes two `array_position` differences from Spark 4.2.0:
   
   | Query | Before | Spark 4.2.0 |
   |---|---|---|
   | `array_position(array(map(1, 1)), map(1L, 1L))` | 
`ARRAY_FUNCTION_DIFF_TYPES` | `INVALID_ORDERING_TYPE` on `"MAP<BIGINT, 
BIGINT>"` |
   | `array_position(array(make_interval(1)), make_interval(2))` | `1` | 
`INVALID_ORDERING_TYPE` on `"INTERVAL"` |
   
   ## What changes are included in this PR?
   
   - New `function/type_coercion.rs` with:
     - `tightest_common_type`: Spark's `findTightestCommonType`. It takes an 
ANSI flag, because `AnsiTypeCoercion` widens an integer and a FLOAT to DOUBLE.
     - `complex_type`: Spark's `findTypeForComplex`. It now widens map keys and 
values, where the old copy required identical entry types.
     - `is_orderable`: Spark's `OrderUtils.isOrderable`. It now also rejects 
`Interval(MonthDayNano)`, which is Spark's `CalendarIntervalType`.
     - `spark_sql_type`: the names Spark's `toSQLType` gives, for error 
messages.
   - `float_normalize` moves from `function/array/` to `function/`, so 
functions outside `array` can use it.
   - `array_position` uses the shared module and passes `ansi_mode = false`, 
because Spark's `ArrayPosition` calls the non-ANSI `TypeCoercion`. Its errors 
now name Spark types and end with the SQLSTATE, as Spark's do. It keeps its own 
check that rejects structs with different field types. That check comes from 
Spark's non-ANSI implicit cast, not from the common-type rules.
   - For two string or binary layouts, `array_position` now picks the layout 
that holds both (for example `LargeUtf8` over `Utf8`) instead of the left one. 
Spark has one string type, so this has no effect a user can see.
   
   The wider-type functions (`findWiderTypeWithoutStringPromotion`, 
`findWiderTypeForDecimal`) are left out of this PR because nothing on `main` 
calls them yet. #332 adds them when it moves to this module.
   
   ## Are these changes tested?
   
   Yes. There are unit tests in `type_coercion.rs`, and new `.slt` cases in 
`array/array_position.slt` cover map widening, flat and nested intervals, and 
the full error text. I checked them against Spark 4.2.0 with the PySpark 
runner. The runner flags two records, and both mismatches were already there on 
`main`: line 129 (Spark's 0-based `[]`) and line 308 (struct names that differ 
only in case, already documented in the file).
   
   ## Are there any user-facing changes?
   
   `array_position` error messages change as described above, and it rejects 
INTERVAL elements. There are no API changes.
   
   This PR was prepared with AI assistance (Claude Code) and reviewed by me.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to