This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch branch-0.16
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/branch-0.16 by this push:
new 68180e86e changelog (#4260)
68180e86e is described below
commit 68180e86e7dfcf7eb90438852a72f6f323472c4f
Author: Andy Grove <[email protected]>
AuthorDate: Thu May 7 18:33:54 2026 -0600
changelog (#4260)
---
dev/changelog/0.16.0.md | 190 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 190 insertions(+)
diff --git a/dev/changelog/0.16.0.md b/dev/changelog/0.16.0.md
new file mode 100644
index 000000000..1e0aea5bd
--- /dev/null
+++ b/dev/changelog/0.16.0.md
@@ -0,0 +1,190 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# DataFusion Comet 0.16.0 Changelog
+
+This release consists of 127 commits from 17 contributors. See credits at the
end of this changelog for more information.
+
+**Fixed bugs:**
+
+- fix: report task output metrics in Spark UI
[#3999](https://github.com/apache/datafusion-comet/pull/3999) (0lai0)
+- fix: cast to and from timestamp_ntz
[#4008](https://github.com/apache/datafusion-comet/pull/4008) (parthchandra)
+- fix: support to_json on Spark 4.0
[#4036](https://github.com/apache/datafusion-comet/pull/4036) (andygrove)
+- fix: enable arrays_overlap
[#3901](https://github.com/apache/datafusion-comet/pull/3901) (kazuyukitanimura)
+- fix: Iceberg reflection for current() on TableOperations hierarchy
[#3895](https://github.com/apache/datafusion-comet/pull/3895) (karuppayya)
+- fix: fall back to Spark for shuffle/sort/aggregate on non-default collated
strings [Spark 4] [#4035](https://github.com/apache/datafusion-comet/pull/4035)
(andygrove)
+- fix: scalar subquery pushdown and reuse for CometNativeScanExec
(SPARK-43402) [#4053](https://github.com/apache/datafusion-comet/pull/4053)
(mbutrovich)
+- fix: fall back for shredded Variant scans on Spark 4.0
[#4084](https://github.com/apache/datafusion-comet/pull/4084) (andygrove)
+- fix: enable Spark 4 SQL tests previously ignored for issues #3313 and #3314
[#4092](https://github.com/apache/datafusion-comet/pull/4092) (andygrove)
+- fix: fall back to Spark for hash join and sort-merge join on non-default
collated string keys [Spark 4]
[#4095](https://github.com/apache/datafusion-comet/pull/4095) (0lai0)
+- fix: reject string/binary read as numeric in native_datafusion scan
[#4091](https://github.com/apache/datafusion-comet/pull/4091) (andygrove)
+- fix: reject incompatible decimal precision/scale in native_datafusion scan
[#4090](https://github.com/apache/datafusion-comet/pull/4090) (andygrove)
+- fix: throw SchemaColumnConvertNotSupportedException from native_datafusion
schema mismatch [#4117](https://github.com/apache/datafusion-comet/pull/4117)
(andygrove)
+- fix: substring with negative start index
[#4017](https://github.com/apache/datafusion-comet/pull/4017) (kazuyukitanimura)
+- fix: honor strictFloatingPoint in RangePartitioning
[#4167](https://github.com/apache/datafusion-comet/pull/4167) (0lai0)
+- fix: [Spark 4.1.1] preserve stored allowDecimalPrecisionLoss in
DecimalPrecision rule
[#4179](https://github.com/apache/datafusion-comet/pull/4179) (andygrove)
+- fix: [Spark 4.1.1] preserve parent struct nullness when all requested fields
missing in Parquet
[#4190](https://github.com/apache/datafusion-comet/pull/4190) (andygrove)
+- fix: support Spark 4.1 BloomFilter V2 format and bit-scattering
[#4196](https://github.com/apache/datafusion-comet/pull/4196) (andygrove)
+- fix: JNI local reference cleanup in JVMClasses::with_env
[#4225](https://github.com/apache/datafusion-comet/pull/4225) (0lai0)
+- fix: broadcast exchange bypasses AQE partition coalescing
[#4163](https://github.com/apache/datafusion-comet/pull/4163) (andygrove)
+- fix: resolve Scala compiler warnings for auto-tupling and bare try
[#4227](https://github.com/apache/datafusion-comet/pull/4227) (andygrove)
+- fix: [Spark 4.1] preserve union output partitioning in CometUnionExec
[#4207](https://github.com/apache/datafusion-comet/pull/4207) (andygrove)
+- fix: re-enable tests skipped for Spark 4.1 (issue #4098)
[#4253](https://github.com/apache/datafusion-comet/pull/4253) (andygrove)
+- fix: cargo clean before release build to avoid stale native libs
[#4257](https://github.com/apache/datafusion-comet/pull/4257) (andygrove)
+
+**Performance related:**
+
+- perf: avoid redundant columnar shuffle when both parent and child are
non-Comet [#4010](https://github.com/apache/datafusion-comet/pull/4010)
(andygrove)
+- perf: reduce per-node allocations in to_native_metric_node
[#4075](https://github.com/apache/datafusion-comet/pull/4075) (andygrove)
+
+**Implemented enhancements:**
+
+- feat: enable native Iceberg reader by default
[#3819](https://github.com/apache/datafusion-comet/pull/3819) (andygrove)
+- feat: support `collect_set`
[#3954](https://github.com/apache/datafusion-comet/pull/3954) (comphead)
+- feat: non-AQE DPP for native Parquet scans, broadcast exchange reuse for DPP
subqueries [#4011](https://github.com/apache/datafusion-comet/pull/4011)
(mbutrovich)
+- feat: add support for array_position expression
[#3172](https://github.com/apache/datafusion-comet/pull/3172) (andygrove)
+- feat: Cast string to timestamp_ntz
[#4034](https://github.com/apache/datafusion-comet/pull/4034) (parthchandra)
+- feat: Add TimestampNTZType support for unix_timestamp
[#4039](https://github.com/apache/datafusion-comet/pull/4039) (parthchandra)
+- feat: fix array_compact for Spark 4.0 and correct return type metadata
[#3796](https://github.com/apache/datafusion-comet/pull/3796) (andygrove)
+- feat: task-level input metrics (bytesRead) for Iceberg native scan
[#4128](https://github.com/apache/datafusion-comet/pull/4128) (mbutrovich)
+- feat: add MapSort expression support for Spark 4.0
[#4076](https://github.com/apache/datafusion-comet/pull/4076) (andygrove)
+- feat: Support Spark expression `str_to_map`
[#3654](https://github.com/apache/datafusion-comet/pull/3654) (unknowntpo)
+- feat: add support for timestamp_seconds expression
[#3146](https://github.com/apache/datafusion-comet/pull/3146) (andygrove)
+- feat: add config to gate converting Spark shuffle to Comet shuffle when
child is non-Comet plan
[#4166](https://github.com/apache/datafusion-comet/pull/4166) (andygrove)
+- feat: AQE DPP for native Parquet scans with broadcast reuse
[#4112](https://github.com/apache/datafusion-comet/pull/4112) (mbutrovich)
+- feat: support regular BuildRight+LeftAnti hash join
[#4073](https://github.com/apache/datafusion-comet/pull/4073) (viirya)
+- feat: add bug-triage Claude skill
[#4109](https://github.com/apache/datafusion-comet/pull/4109) (andygrove)
+- feat: support `PartialMerge` aggregation mode
[#4003](https://github.com/apache/datafusion-comet/pull/4003) (comphead)
+- feat: add encode time tracking for shuffle operations
[#4068](https://github.com/apache/datafusion-comet/pull/4068) (0lai0)
+- feat: Add support for Spark ToDegrees and ToRadians math expressions
[#3786](https://github.com/apache/datafusion-comet/pull/3786) (rafafrdz)
+- feat: Add support for Spark Acosh, Asinh, Atanh math expressions
[#3787](https://github.com/apache/datafusion-comet/pull/3787) (rafafrdz)
+- feat: Add support for Spark Cbrt math expression
[#3788](https://github.com/apache/datafusion-comet/pull/3788) (rafafrdz)
+- feat: Add support for Spark Pi math expression
[#3789](https://github.com/apache/datafusion-comet/pull/3789) (rafafrdz)
+- feat: support Parquet field ID matching in native_datafusion scan
[#4216](https://github.com/apache/datafusion-comet/pull/4216) (mbutrovich)
+- feat: support AQE DPP broadcast reuse for Iceberg native scans
[#4215](https://github.com/apache/datafusion-comet/pull/4215) (mbutrovich)
+- feat: add support for url_encode, url_decode, and try_url_decode
[#4231](https://github.com/apache/datafusion-comet/pull/4231) (parthchandra)
+- feat: support TimestampType join keys in SortMergeJoin
[#3986](https://github.com/apache/datafusion-comet/pull/3986) (andygrove)
+
+**Documentation updates:**
+
+- docs: Add changelog for 0.15.0
[#4000](https://github.com/apache/datafusion-comet/pull/4000) (andygrove)
+- docs: Update README and benchmark results for 0.15.0 release
[#3995](https://github.com/apache/datafusion-comet/pull/3995) (andygrove)
+- docs: fix errors in benchmark pages
[#4001](https://github.com/apache/datafusion-comet/pull/4001) (andygrove)
+- docs: split compatibility guide into multiple pages
[#4055](https://github.com/apache/datafusion-comet/pull/4055) (andygrove)
+- docs: Generate expression compatibility docs from code
[#4057](https://github.com/apache/datafusion-comet/pull/4057) (andygrove)
+- doc: update documentation for cast and datetime functions
[#4058](https://github.com/apache/datafusion-comet/pull/4058) (parthchandra)
+- docs: add compatibility documentation to all expressions
[#4067](https://github.com/apache/datafusion-comet/pull/4067) (andygrove)
+- docs: rename SQL File Tests to Comet SQL Tests
[#4108](https://github.com/apache/datafusion-comet/pull/4108) (andygrove)
+- docs: add Understanding Comet Plans user guide page
[#4086](https://github.com/apache/datafusion-comet/pull/4086) (andygrove)
+- docs: support conditional content for snapshot vs release builds
[#4030](https://github.com/apache/datafusion-comet/pull/4030) (andygrove)
+- docs: update Spark version support and add version compatibility page
[#4138](https://github.com/apache/datafusion-comet/pull/4138) (andygrove)
+- docs: improve review skill and contributor guide for serde patterns
[#4132](https://github.com/apache/datafusion-comet/pull/4132) (andygrove)
+- docs: Fix errors in list of supported Spark versions
[#4141](https://github.com/apache/datafusion-comet/pull/4141) (andygrove)
+- docs: Update roadmap in contributor guide
[#4144](https://github.com/apache/datafusion-comet/pull/4144) (andygrove)
+- docs: add implement-comet-expression Claude skill
[#4158](https://github.com/apache/datafusion-comet/pull/4158) (andygrove)
+- docs: add roadmap items for spillable hash join, UDF support, memory
management, and 1.0.0
[#4171](https://github.com/apache/datafusion-comet/pull/4171) (andygrove)
+- docs: start Spark 4.1 known-limitations section, seeded with #4199
[#4202](https://github.com/apache/datafusion-comet/pull/4202) (andygrove)
+- docs: document Spark 4 IntelliJ setup
[#4198](https://github.com/apache/datafusion-comet/pull/4198) (yuboxx)
+- docs: refresh Gluten comparison with ANSI, Spark 4, and Iceberg coverage
[#4169](https://github.com/apache/datafusion-comet/pull/4169) (andygrove)
+- docs: check off 53 implemented expressions in support doc
[#4147](https://github.com/apache/datafusion-comet/pull/4147) (andygrove)
+- docs: replace project logos with updated branding
[#4230](https://github.com/apache/datafusion-comet/pull/4230) (pranamya123)
+- docs: Documentation updates in preparation for 0.16 release
[#4244](https://github.com/apache/datafusion-comet/pull/4244) (andygrove)
+
+**Other:**
+
+- chor: enable array_distinct
[#3987](https://github.com/apache/datafusion-comet/pull/3987) (kazuyukitanimura)
+- chore: Improve shuffle fallback logic
[#3989](https://github.com/apache/datafusion-comet/pull/3989) (andygrove)
+- chore: Start 0.16 development
[#4016](https://github.com/apache/datafusion-comet/pull/4016) (andygrove)
+- chore: update compatibility guide for primitive to string casts
[#4012](https://github.com/apache/datafusion-comet/pull/4012) (parthchandra)
+- test: add sql-file test confirming fallback on parquet variant reads
[#4021](https://github.com/apache/datafusion-comet/pull/4021) (andygrove)
+- chore: skip Iceberg and Spark SQL test workflows on test-only changes
[#4023](https://github.com/apache/datafusion-comet/pull/4023) (andygrove)
+- ci: force GNU ld on Linux to fix -ljvm linker error
[#4024](https://github.com/apache/datafusion-comet/pull/4024) (andygrove)
+- chore: update pending PR filter
[#4025](https://github.com/apache/datafusion-comet/pull/4025) (comphead)
+- test: run more Spark 4 tests
[#4013](https://github.com/apache/datafusion-comet/pull/4013) (kazuyukitanimura)
+- chore: update documentation links for 0.15.0 release
[#4029](https://github.com/apache/datafusion-comet/pull/4029) (andygrove)
+- test: run more JVM tests
[#4026](https://github.com/apache/datafusion-comet/pull/4026) (kazuyukitanimura)
+- feature: Support Spark expression: arrays_zip
[#3643](https://github.com/apache/datafusion-comet/pull/3643) (hsiang-c)
+- test: fix SparkToColumnar plan-shape assertions on Spark 4
[#4032](https://github.com/apache/datafusion-comet/pull/4032) (andygrove)
+- test: unignore DynamicPartitionPruning static scan metrics test
[#4038](https://github.com/apache/datafusion-comet/pull/4038) (andygrove)
+- test: add non-AQE DPP edge case coverage for native Parquet scans
[#4037](https://github.com/apache/datafusion-comet/pull/4037) (mbutrovich)
+- test: unignore passing Spark 4.0 #3321 tests, retag remaining failures
[#4041](https://github.com/apache/datafusion-comet/pull/4041) (andygrove)
+- test: unignore passing DPP test, retag remaining failures
[#4046](https://github.com/apache/datafusion-comet/pull/4046) (andygrove)
+- chore: enable array_union
[#4043](https://github.com/apache/datafusion-comet/pull/4043) (kazuyukitanimura)
+- test: add date_trunc DST regression test in non-UTC session timezone
[#4040](https://github.com/apache/datafusion-comet/pull/4040) (andygrove)
+- ci: skip heavy test workflows for GenerateDocs and changelog changes
[#4056](https://github.com/apache/datafusion-comet/pull/4056) (andygrove)
+- chore(deps): bump the all-other-cargo-deps group in /native with 3 updates
[#4061](https://github.com/apache/datafusion-comet/pull/4061) (dependabot[bot])
+- ci: add concurrency blocks to more workflows to cancel on new commit
[#4064](https://github.com/apache/datafusion-comet/pull/4064) (mbutrovich)
+- chore: gitignore generated docs directories
[#4066](https://github.com/apache/datafusion-comet/pull/4066) (andygrove)
+- chore: promote array_insert to compatible and consolidate expression docs
[#4065](https://github.com/apache/datafusion-comet/pull/4065) (andygrove)
+- test: re-enable sql_hive-1 for Spark 4.0 and fix two small failures
[#4047](https://github.com/apache/datafusion-comet/pull/4047) (andygrove)
+- ci: stopping running PR builds with 3.5.5/3.5.6
[#4103](https://github.com/apache/datafusion-comet/pull/4103) (andygrove)
+- chore: Bump Spark 4.0.1 to 4.0.2
[#4114](https://github.com/apache/datafusion-comet/pull/4114) (andygrove)
+- build: add spark-4.1 Maven profile and shim sources
[#4097](https://github.com/apache/datafusion-comet/pull/4097) (andygrove)
+- refactor: consolidate identical spark-4.0 and spark-4.1 shims into spark-4.x
[#4118](https://github.com/apache/datafusion-comet/pull/4118) (andygrove)
+- ci: enable Spark 4.1 PR test matrix
[#4106](https://github.com/apache/datafusion-comet/pull/4106) (andygrove)
+- build: add spark-4.2 Maven profile targeting 4.2.0-preview4
[#4119](https://github.com/apache/datafusion-comet/pull/4119) (andygrove)
+- ci: reduce macOS PR matrix to single Spark 4.0 profile
[#4104](https://github.com/apache/datafusion-comet/pull/4104) (andygrove)
+- chore: audit array_intersect and expand SQL test coverage
[#4071](https://github.com/apache/datafusion-comet/pull/4071) (andygrove)
+- chore: support Spark 4.1/4.2 for golden files generation
[#4134](https://github.com/apache/datafusion-comet/pull/4134) (comphead)
+- ci: add Spark 4.0 / JDK 21 profile
[#4060](https://github.com/apache/datafusion-comet/pull/4060) (james-willis)
+- ci: Enable Comet PR test matrix and TPCDS plan-stability for Spark 4.2
[#4126](https://github.com/apache/datafusion-comet/pull/4126) (andygrove)
+- test: support fallback chain in CometPlanStabilitySuite, dedupe existing
goldens [#4129](https://github.com/apache/datafusion-comet/pull/4129)
(andygrove)
+- chore: Update `spark_expressions_support.md` doc
[#4165](https://github.com/apache/datafusion-comet/pull/4165) (comphead)
+- build: Enable Spark SQL tests for Spark 4.1.1
[#4093](https://github.com/apache/datafusion-comet/pull/4093) (andygrove)
+- test: unignore InjectRuntimeFilterSuite tests gated on issue #242
[#4178](https://github.com/apache/datafusion-comet/pull/4178) (andygrove)
+- chore(deps): bump cc from 1.2.60 to 1.2.61 in /native in the
all-other-cargo-deps group
[#4168](https://github.com/apache/datafusion-comet/pull/4168) (dependabot[bot])
+- test: [Spark 4.1.1] unignore SPARK-52921 union partitioning tests
[#4195](https://github.com/apache/datafusion-comet/pull/4195) (andygrove)
+- test: [Spark 4.1.1] unignore CachedBatchSerializerNoUnwrapSuite
[#4204](https://github.com/apache/datafusion-comet/pull/4204) (andygrove)
+- test: relax bytesRead ratio assertion for Spark 4.1+
[#4197](https://github.com/apache/datafusion-comet/pull/4197) (andygrove)
+- deps: Bump OpenDAL to 0.56.0
[#4217](https://github.com/apache/datafusion-comet/pull/4217) (mbutrovich)
+- ci: pin JDK per Spark version in Iceberg workflow matrix
[#4120](https://github.com/apache/datafusion-comet/pull/4120) (dmefs)
+- test: skip flaky StateStoreSuite under Comet and disambiguate JDK matrix
names [#4226](https://github.com/apache/datafusion-comet/pull/4226) (andygrove)
+- Map ProfileCredentialsProvider to profile credential chain
[#4213](https://github.com/apache/datafusion-comet/pull/4213) (karuppayya)
+- test: add explicit negative-count cases for array_repeat
[#4182](https://github.com/apache/datafusion-comet/pull/4182) (slavlotski)
+- test: add unhex dictionary coverage
[#4222](https://github.com/apache/datafusion-comet/pull/4222) (yuboxx)
+- chore: Enable GitHub project view
[#4247](https://github.com/apache/datafusion-comet/pull/4247) (andygrove)
+- test: dedupe redundant Spark 4.2 TPC-DS plan-stability golden files
[#4243](https://github.com/apache/datafusion-comet/pull/4243) (andygrove)
+- build: change default Maven profile to Spark 4.1 / Scala 2.13
[#4140](https://github.com/apache/datafusion-comet/pull/4140) (andygrove)
+- chore: remove legacy ENABLE_COMET_SCAN_ONLY and ENABLE_COMET_ANSI_MODE env
vars from Spark diffs
[#4252](https://github.com/apache/datafusion-comet/pull/4252) (parthchandra)
+
+## Credits
+
+Thank you to everyone who contributed to this release. Here is a breakdown of
commits (PRs merged) per contributor.
+
+```
+ 78 Andy Grove
+ 9 Matt Butrovich
+ 7 Parth Chandra
+ 6 KAZUYUKI TANIMURA
+ 5 ChenChen Lai
+ 5 Oleks V
+ 4 Rafael Fernández
+ 2 Karuppayya
+ 2 Yubo Xu
+ 2 dependabot[bot]
+ 1 Eric Chang
+ 1 James Willis
+ 1 Kuo-Hao Huang
+ 1 Liang-Chi Hsieh
+ 1 Pranamya Vadlamani
+ 1 Vladislav Zabolotsky
+ 1 hsiang-c
+```
+
+Thank you also to everyone who contributed in other ways such as filing
issues, reviewing PRs, and providing feedback on this release.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
