andygrove opened a new pull request, #345: URL: https://github.com/apache/datafusion-spark/pull/345
## Which issue does this PR close? - Relates to #332, #331, #330 and #70, which each carry their own copy of these rules. Requested in [this review comment](https://github.com/apache/datafusion-spark/pull/332#discussion_r4602650086). ## Rationale for this change Functions that need one type for several arguments (`array_position` now, `greatest`, `least`, `array_remove`, `array_max`, `array_min` and `array_insert` in open PRs) each carry a copy of Spark's common-type rules. The copies have already drifted apart. This PR gives them one shared copy, so fixes land once. It also fixes two `array_position` differences from Spark 4.2.0: | Query | Before | Spark 4.2.0 | |---|---|---| | `array_position(array(map(1, 1)), map(1L, 1L))` | `ARRAY_FUNCTION_DIFF_TYPES` | `INVALID_ORDERING_TYPE` on `"MAP<BIGINT, BIGINT>"` | | `array_position(array(make_interval(1)), make_interval(2))` | `1` | `INVALID_ORDERING_TYPE` on `"INTERVAL"` | ## What changes are included in this PR? - New `function/type_coercion.rs` with: - `tightest_common_type`: Spark's `findTightestCommonType`. It takes an ANSI flag, because `AnsiTypeCoercion` widens an integer and a FLOAT to DOUBLE. - `complex_type`: Spark's `findTypeForComplex`. It now widens map keys and values, where the old copy required identical entry types. - `is_orderable`: Spark's `OrderUtils.isOrderable`. It now also rejects `Interval(MonthDayNano)`, which is Spark's `CalendarIntervalType`. - `spark_sql_type`: the names Spark's `toSQLType` gives, for error messages. - `float_normalize` moves from `function/array/` to `function/`, so functions outside `array` can use it. - `array_position` uses the shared module and passes `ansi_mode = false`, because Spark's `ArrayPosition` calls the non-ANSI `TypeCoercion`. Its errors now name Spark types and end with the SQLSTATE, as Spark's do. It keeps its own check that rejects structs with different field types. That check comes from Spark's non-ANSI implicit cast, not from the common-type rules. - For two string or binary layouts, `array_position` now picks the layout that holds both (for example `LargeUtf8` over `Utf8`) instead of the left one. Spark has one string type, so this has no effect a user can see. The wider-type functions (`findWiderTypeWithoutStringPromotion`, `findWiderTypeForDecimal`) are left out of this PR because nothing on `main` calls them yet. #332 adds them when it moves to this module. ## Are these changes tested? Yes. There are unit tests in `type_coercion.rs`, and new `.slt` cases in `array/array_position.slt` cover map widening, flat and nested intervals, and the full error text. I checked them against Spark 4.2.0 with the PySpark runner. The runner flags two records, and both mismatches were already there on `main`: line 129 (Spark's 0-based `[]`) and line 308 (struct names that differ only in case, already documented in the file). ## Are there any user-facing changes? `array_position` error messages change as described above, and it rejects INTERVAL elements. There are no API changes. This PR was prepared with AI assistance (Claude Code) and reviewed by me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
