andygrove opened a new pull request, #2241: URL: https://github.com/apache/datafusion-ballista/pull/2241
# Which issue does this PR close? Closes #1920. Supersedes #1921, which targeted `main` from a personal topic branch. This PR opens the long-lived `datafusion-55` branch instead, so all contributors can send PRs against it. # Rationale for this change Ballista depends on published DataFusion crates.io releases (today, `54`). Because a full release cycle passes between DataFusion versions, breaking changes only surface in Ballista at upgrade time, all at once, making each upgrade large and risky. This branch switches the `datafusion*` crates to a pinned git revision of `apache/datafusion` `main`. Tracking unreleased DataFusion continuously lets us catch breaking API and behavior changes in small increments, close to when they land upstream, rather than accumulating them until the next release. `datafusion-55` is a long-lived branch. The pinned rev gets bumped periodically so DataFusion regressions surface early, and contributors send their DataFusion-55 porting work as PRs against `datafusion-55` rather than against `main`. The branch merges into `main` when DataFusion 55 is released. # What changes are included in this PR? Pin the `datafusion*` crates to a git `rev` of `apache/datafusion` `main` (currently `70c26a06`) and bump `arrow`/`arrow-flight` to `59` to match. Port to the DataFusion `main` APIs: - `TableReference` moved from `datafusion::sql` to `datafusion::common`. - `PhysicalExtensionCodec::try_decode`/`try_encode` gained a `PhysicalProtoConverterExtension` argument. - `MetricValue` gained a `PeakMemoryUsage` variant; serialize it as a named gauge. - `Partitioning` gained a `Range` variant; format it in the DOT graph. - `EnforceSorting` and `EnforceDistribution` were unified into `EnsureRequirements`; use it for post-stage-split re-optimization. - `ExecutionPlan::partition_statistics` is deprecated in favour of `StatisticsContext::compute`; migrate all call sites. This is a **correctness fix**: DataFusion operators that no longer override the deprecated method return unknown statistics through it, which would silently disable Ballista's broadcast-threshold checks. - `ExecutionPlan::required_input_distribution` is deprecated in favour of `input_distribution_requirements`. - `QueryPlanner::create_physical_plan` takes `&dyn Session` instead of `&SessionState`. - `ListingOptions` dropped its `target_partitions`/`collect_stat` fields; build it via the builder (`target_partitions` now comes from the session config). - `Distribution::HashPartitioned` is deprecated; use `KeyPartitioned`. - `datafusion_substrait::serializer::deserialize_bytes` is now synchronous and takes `&[u8]`. Disable dynamic filter pushdown in the AQE mock session config to match the Ballista production default, so join-input swaps during re-optimization stay legal. Sync the vendored `datafusion_common.proto` for the new `max_in_list_size` field, and refresh the EXPLAIN ANALYZE and AQE plan snapshots for DataFusion's dropped `peak_mem_used` metric, its new `RightSemi` join lowering, and its `null_aware` join display. # Are there any user-facing changes? Ballista on this branch builds against a git revision of DataFusion `main` rather than a crates.io release. There are no changes to Ballista's own public API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
