comphead opened a new pull request, #6927: URL: https://github.com/apache/datafusion-comet/pull/6927
## Which issue does this PR close? Closes #6909. ## Rationale for this change Spark plans `SortAggregateExec` for `min`/`max` over `STRING` or `BINARY`, because the aggregation buffer holds the input value and `HashAggregateExec` needs fixed-width buffers. #4565 made `SortAggregateExec` native, but `CometMin` and `CometMax` still declined these types, so the whole aggregate fell back with `Unsupported data type: StringType`, together with the sorts and row conversions around it. String `first`/`last` in the same plan shape already run natively. ## What changes are included in this PR? - `AggSerde.minMaxSupportLevel` accepts `StringType` and `BinaryType`. Spark compares both as unsigned bytes (`ByteArray.compareBinary`, on 3.4 through 4.1), and DataFusion's `min`/`max` compare raw bytes too (`MinMaxBytesAccumulator`). The native planner already sends these types to DataFusion's `min_udaf`/`max_udaf`, so there is no native change. - A string with a non-default collation (Spark 4.0+) still falls back, with the reason from `ComparisonUtils.nonDefaultCollationReason`, because Spark orders it by its collation. - `minMaxDataTypeSupported` is unchanged, so window `min`/`max` and `max_by`/`min_by` keep their current support. The operators compatibility guide now says that string and binary `min`/`max` run natively as plain aggregations but not as window functions. ## How are these changes tested? - `min_max.sql`: the global aggregate that expected the fallback now has to run natively. New queries cover grouped and global aggregates over string and binary columns: prefixes, case, empty values, an all-NULL group, empty input, characters whose UTF-8 order differs from their UTF-16 order, and bytes on both sides of `X'7F'`/`X'80'`. One more query puts `min`/`max` next to `collect_set`, which Spark plans as `ObjectHashAggregateExec`. The file now runs with and without Parquet dictionary encoding. - `min_max_collation.sql` (Spark 4.0+): `min`/`max` over a `UTF8_LCASE` string must fall back. The Spark 4.1 profile compiles locally with scalastyle and spotless clean. The SQL file tests have not run locally yet, so CI is their first run. Two Spark SQL tests assert plan shapes with a string `max`: SPARK-31620 in `DataFrameAggregateSuite` and the randomized tests in the Hive `ObjectHashAggregateSuite`. Both read `VALUES` or RDD input, which stays on Spark, so they should be unaffected. `run-spark-4.1-tests` is applied to confirm. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
