SEPURI-SAI-KRISHNA commented on code in PR #59135: URL: https://github.com/apache/spark/pull/59135#discussion_r4284801866
########## docs/sql-migration-guide.md: ########## @@ -48,6 +48,7 @@ license: | - Since Spark 4.4, `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` defaults to `true`. A V2 scan that reports a keyed partitioning but no explicit ordering now also reports itself sorted by its partition key expressions, because every row of such a partition evaluates them to the same value. Partition transforms such as `days(ts)` or `bucket(8, id)` are left out of that ordering. This lets Spark drop a `Sort` it would otherwise place above the scan. An aggregate that groups by a leading run of the other key expressions, in key order, may also be planned as a sort aggregate rather than a hash aggregate. This is because `spark.sql.execution.replaceHashWithSortAgg` keys off the reported ordering. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` to `false`. - Since Spark 4.4, `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` defaults to `true`. When `GroupPartitionsExec` merges several input partitions that share one partition key value into a single output partition, it now keeps sort orders over the partition key expressions in the ordering it reports, since those expressions are constant within the merged partition. Sort orders over partition transforms such as `days(ts)` or `bucket(8, id)` are left out. Orders over other columns are still dropped, as the concatenation invalidates them, and a join that reduced the partition keys onto a common key space (see `spark.sql.sources.v2.bucketing.allowCompatibleTransforms.enabled`) reports no order, since the merged partitions then share only the reduced key. This can remove a `Sort` downstream of the merge. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` to `false`. - Since Spark 4.4, the value columns of `UNPIVOT` and `Dataset.unpivot` merge the nested nullability (array element, map value, and struct field nullability) of all unpivoted values. Previously, a value column took the nested nullability of the first listed value, so a `NULL` array element, map value, or struct field of a later value could be returned as a non-`NULL` value such as `0`, and `IS NULL` on it returned `false`. A view created by an earlier Spark version over such an `UNPIVOT` may have recorded the non-nullable nested type in its schema and can fail to read; recreate it with `CREATE OR REPLACE VIEW`. To restore the previous behavior, set `spark.sql.legacy.unpivot.useFirstValueNestedNullability` to `true`. +- Since Spark 4.4, `date_add`, `date_sub`, `dateadd`, and adding or subtracting an integer number of days to or from a date (for example `date + 1` or `date - 1`) fail with `DATETIME_OVERFLOW` when the number of days since 1970-01-01 in the result does not fit in an `INT`, regardless of `spark.sql.ansi.enabled`. Previously, the result silently wrapped around; for example, `date_add(DATE'2020-01-01', 2147483647)` returned a date in year -5877591. `try_add` and `try_subtract` on dates now return `NULL` in this case instead of the wrapped date. Review Comment: Thanks, good catch. I checked it: before this change that query counted 0 rows in the frame instead of 1, and now it fails with `DATETIME_OVERFLOW` with ANSI mode on and off. A `PRECEDING` offset of 2147483647 still works, and so does a DAY-interval offset. I added a sentence on DATE-ordered `RANGE` frames with an integer offset to the migration guide entry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
