andygrove commented on issue #5327: URL: https://github.com/apache/datafusion-comet/issues/5327#issuecomment-5637162631
## What is already on `main` for 1.1.0 218 commits have landed on `main` since `branch-1.0` was cut at af534e0 (74 `fix`, 45 `feat`, 21 `perf`, plus chores, tests, docs and CI). Highlights: **Dependencies** - DataFusion 55.0 and Arrow/Parquet 59.2 (#5262). Follow-ups tracked in #5709. **New native support** - Celeborn remote shuffle: native map-side push, reader and planning (#5473 through #5537). - Native Iceberg V2 writer via iceberg-rust, with split writer/committer operators (#5361, #4658, #5298). Both write flags still default off. - Native dynamic filter pushdown from hash joins into Parquet scans (#5699). - Aggregates: `max_by`/`min_by` (#4817), `mode` (#4782), the `regr_*` family (#4775), `WindowGroupLimitExec` (#4870). - Expressions: `explode_outer` (#5192), `make_interval` (#5039), `spark_sequence` (#5614), `unbase64` (#5451), `concat_ws` with arrays (#5725), Iceberg system functions (#5638), plus codegen-dispatch routing for `encode`, `timestampadd`/`timestampdiff`, `translate`, `to_csv`, `lpad`/`rpad`, `round` on floats and unrecognized `Invoke`/`StaticInvoke`. - Scan: S3-compatible filesystems (#5314), `_metadata` constant columns (#5237), unprojected Spark 4 VARIANT columns (#5377), unicode case-sensitive field names (#5602), scan I/O and read-amplification metrics (#5453). - Experimental in-memory cache, disabled by default (#5051). **Notable correctness fixes** - Exchange reuse returned wrong aggregate results (#5470). - Silent overflow reading Parquet `TIMESTAMP_MILLIS` (#5177, #5740). - Float sort and window rank keys not normalized (#5469); struct null mask ignored when hashing fields (#5754). - Decimal to float/double cast rounding (#5684); float to integral ANSI bounds (#5683); string to timestamp segment rules (#5682). - Native shuffle 2GB task serialization OOM on many partitions (#5392); memory leak after failed Arrow import (#5539). - Memory and spill metrics were wrong or missing across shuffle, aggregate and non-shuffle stages (#5370, #5408, #5445, #5463, #5497, #5516, #5554, #5628). **Performance** - `collect_list`/`collect_set` GroupsAccumulator (#5803), explode slicing instead of gathering (#5667), shuffle scratch and IPC context reuse (#5038, #5568), faster ArrowWriter (#5046, #5442), vectorized integer-to-decimal cast (#4939), `spark_floor` (#4911), `spark_base64` (#4885). Full list: `git log af534e0..main`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
