andygrove commented on code in PR #210:
URL: https://github.com/apache/datafusion-site/pull/210#discussion_r4448063521
##########
content/blog/2026-11-03-datafusion-comet-1.2.0.md:
##########
@@ -0,0 +1,523 @@
+---
+layout: post
+title: Apache DataFusion Comet 1.2.0 Release
+date: 2026-11-03
+author: pmc
+categories: [subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Apache DataFusion PMC is pleased to announce version 1.2.0 of the
[Comet](https://datafusion.apache.org/comet/) subproject.
+
+Comet is an accelerator for Apache Spark that translates Spark physical plans
to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.
+
+This release covers roughly five weeks of development since 1.1.0: 399 commits
from 40 contributors. See the
+[change log] for the full list of changes. Unless stated otherwise,
performance figures come from the
+benchmarks in the linked PRs.
+
+[change log]:
https://github.com/apache/datafusion-comet/blob/branch-1.2/docs/source/changelog/1.2.0.md
+
+Comet has always aimed to return the same results as Spark. Much of this cycle
went into the edge cases that
+remained, working through whole categories at a time (timezones,
floating-point `-0.0` and `NaN`, collations)
+and adding tests that keep them fixed.
+
+The highlights:
+
+- **Results that match Spark**: timezone labels, `-0.0` and `NaN`, collations,
decimal `AVG`, nested types,
+ broadcasts, and join order.
+- **Memory, spill, and shuffle at scale**: memory returned to Spark sooner,
fewer memory-related task failures,
+ much smaller JVM shuffle memory use with many partitions, and smaller
shuffle files.
+- **Iceberg**: native writes are a documented, opt-in feature with support for
format-version 3 tables, and the
+ native scan reads fewer nested fields and more files at once.
+- **More of the plan runs natively**: existence joins, `SortAggregateExec`,
the `PIVOT` fast path, and
+ HyperLogLog sketch functions, among others.
+- **New places to run**: a native Delta Lake scan (experimental), Arrow Python
UDFs (experimental), and Spark 4.2
+ as a supported version.
+
+## Results That Match Spark
+
+### Timestamps and timezones
+
+Spark labels a timestamp with the session timezone, and Comet's native code
has to produce the same label, not
+only the same instant. Several native expressions got this wrong. 1.2.0 fixes
them:
+
+- `timestamp_seconds` results are labeled as UTC timestamps ([#6341]),
`date_trunc` keeps its input's label
+ ([#6345]), and `CASE` and `COALESCE` relabel timestamp branches instead of
panicking ([#6347]).
+- Session timezone IDs are normalized before they reach native code ([#6351]).
+- `date_trunc` with a format that differs from row to row uses the same kernel
as a literal format ([#6354]).
+- The driver warns when the native and JVM timezone databases differ
([#6353]), because Comet cannot correct
+ for that difference.
+- The test harness now fails a test when a native plan mislabels a timestamp,
so this class of bug can't come
+ back unnoticed ([#6868], [#6930]).
+
+Thanks to [@andygrove] for the timezone sweep and [@LinSimon-901101] for the
label checks. The new
+[timezone handling page] in the contributor guide explains the rules.
+
+[timezone handling page]:
https://datafusion.apache.org/comet/contributor-guide/timezones.html
+
+### Floating point: `-0.0`, `0.0`, and `NaN`
+
+Spark has two rules for these values. Some operations treat `-0.0` and `0.0`
as equal and every `NaN` as one value,
+and others keep them distinct. Which rule an operation follows can change
between Spark versions, and DataFusion
+makes its own choices. The differences show up as wrong joins, groups, and
sort orders. In 1.2.0, these follow
+Spark:
+
+- **Hashing**: `hash`, `xxhash64`, and native shuffle hash `NaN` as Spark does
([#6413]).
+- **Comparison and ordering**: `min`, `max`, `greatest`, and `least`
([#6457]), sort and window keys nested in
+ arrays and structs ([#6475]), and `percentile` for negative `NaN`s ([#6829]).
+- **Array functions**: `array_remove`, `sort_array`, `array_distinct`, and
`array_union` ([#6518], [#6563]), and
+ `arrays_overlap` on the Spark versions where `-0.0` and `0.0` are one value
([#6989]).
+- **Arithmetic**: `signum(-0.0)` ([#6556]) and the zero check in `Remainder`
([#6896]).
+- **Maps**: Comet no longer builds a map natively when its key type holds a
`FLOAT` or `DOUBLE` ([#6992]).
+ Spark finds duplicate keys by value, so `NaN`s with different bits are one
key, and from 4.0 it also
+ normalizes `-0.0` keys to `0.0`. Native code told keys apart by their bits.
+
+The contributor guide now has a [floating point page] that says which Spark
rule each expression follows and how
+to test for it. A sweep suite crosses edge values with every operator context,
so a DataFusion upgrade that
+shifts float behavior will fail a test before it reaches users.
+
+Thanks to [@andygrove] for the float sweep, [@rich7420] for `percentile`, and
[@zhangfengcdt] for `signum`.
+
+[floating point page]:
https://datafusion.apache.org/comet/contributor-guide/floating_point.html
+
+### Collations
+
+Strings with a non-default collation compare differently from ordinary
strings. For collated casts, sort keys, and
+comparisons, Comet now falls back to Spark or runs the expression through
[codegen dispatch] instead of comparing
+bytes ([#5302], [#6206], [#6471], [#6480]). Thanks to [@stantheman0128],
[@LinSimon-901101], and [@viirya].
+
+[codegen dispatch]:
https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/
+
+### Other wrong-results fixes
+
+- **Decimal `AVG`**: results were `NULL` or wrong for sums that overflowed an
intermediate `i128`, and empty and
+ overflowing partial states were mixed up. 1.2.0 repairs the division and the
buffers ([#5420], thanks to
+ [@sunchao]) and falls back to Spark for the cases it cannot yet do correctly
([#7017]).
+- **Variance, covariance, and correlation**: these now update and merge in
Spark's order, so values that are large
+ and close together give Spark's result, and `regr_r2` returns `NULL` or
`DIVIDE_BY_ZERO` where Spark does
+ ([#6076], thanks to [@rich7420]).
+- **Nested values in comparisons and sorts**: `greatest` and `least` compare
struct arguments by position and
+ order nulls inside nested values first ([#6760], [#6761], thanks to
[@viirya]), and native sort keeps Spark's
+ null ordering inside nested values ([#6968], thanks to [@Weijun-H]).
+- **Missing nested Parquet fields**: when every nested field a query asks for
is missing from a Parquet file,
+ Comet could return a non-null struct, array, or map where Spark returns
`NULL`, which changed `IS NULL`, filters,
+ and `count`. Comet now follows Spark's version-dependent rules, and falls
back for nested reads of name-mapped
+ Iceberg tables ([#7085], thanks to [@rich7420]).
+- **Nested binary read as strings**: a nested Parquet `BINARY` value read as a
string that held invalid UTF-8
+ became `NULL`. It is now decoded the way Comet's `CAST(binary AS STRING)`
decodes it, replacing ill-formed bytes
+ with U+FFFD ([#7106], thanks to [@manuzhang]).
+- **Sliced broadcasts**: broadcasts of sliced string, binary, array, or map
batches could return wrong values or
+ fail with `OversizedAllocationException` ([#6900], thanks to [@comphead]).
+- **Identity broadcasts**: a Spark operator that keeps every row of a Comet
broadcast could see the last row of
+ each batch repeated in place of the others. Comet now copies each row
([#7260], thanks to [@pingzh]).
+- **Join output order**: Spark drops a sort above a hash join because the join
keeps its streamed side's order.
+ Comet's native hash join did not keep it for some inputs, so the query could
return unsorted partitions
+ ([#6835], [#6810], thanks to [@dwsmith1983]).
+- **Spark patch releases**: Spark 4.0.5, 4.1.4, and 4.2.1 change what
`sequence` returns when `stop - start`
Review Comment:
We missed on commit when generating the changelog. Too late to fix now for
the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
