NoahKusaba opened a new issue, #2888: URL: https://github.com/apache/datafusion-ballista/issues/2888
I went through the 147 open issues and checked each one against `main` (code search), the PRs that mention it, and its comment thread. Below are the ones that look safe to close, plus a few that need a quick check first. Nothing was rebuilt or benchmarked to confirm these, so please check before closing. cc @milenkovicm, since you asked to be tagged on old issues that can be closed (#2849). ## Already fixed | Issue | Why it can be closed | |---|---| | #2193 NOT IN loses null semantics in SortMergeJoinExec | DataFusion fixed this in apache/datafusion#22810, and `main` is on DataFusion 56. #2188 and #2199 also handle `NOT IN` on the Ballista side. | | #1960 Surface column statistics via ShuffleReaderExec | Done in the #2664 → #2671 stack. #2664 says "Closes #1960" but was closed rather than merged; its commits came in through #2671. `state/aqe/mod.rs` now reads `output_column_stats`. | | #2063 AQE hangs when a replan cancels an in-flight stage | #2192 now removes stages a replan cancels and ignores their late task updates. The "there are stages to be cancelled but its not implemented" warning is gone from `main`. | | #348 Implement broadcast join optimization | #342 (broadcast exchange) is closed, and #1900 and #1904 added broadcast joins. | | #890 Add Iceberg table support | #2638 (Iceberg scans) is merged, and the remaining Iceberg work is tracked in #2174. | | #1151 Push-based shuffle / all-at-once stage scheduling | Covered by the pipelined execution mode: #2657 is merged and the design issue #2652 is closed. | | #408 Bubble execution mode | Same as #1151: covered by pipelined mode (#2652, #2657). | | #332 TaskGroups (share memory between task slots) | #2038 added multi-partition tasks, so one task can run several partitions. | | #406 Control `collect_stat` for the table provider | #796 added the setting, and Ballista no longer overrides DataFusion's `collect_statistics`. | | #119 ExecutorReservation during task scheduling | `ExecutorReservation` was removed in #823. | | #645 Source data cache layer | A 2024 comment notes that its only unchecked item, #833, was merged. | | #2162 S3 prefetching like Spark | The comment on the issue says the benchmark gap behind it is resolved (721s vs 700s compared with Spark 3.4). | | #2367 TPC-H "Failed to open partition file" | The cause was a stale compiled Python extension in the local checkout, not a Ballista bug. The CI follow-up is tracked in #2372. | | #2712 Nightly Docker image build failed | Every nightly Docker build from Nov 12 to Nov 17 succeeded. | | #2727 Run a stage once when several stages read it | #2765 (which closed #1953) runs a repeated subplan only once under AQE, matching stages by a hash of their subtree. It would be good to confirm this covers physical plans that Comet submits. | ## Duplicates or covered by another issue | Issue | Covered by | |---|---| | #553 Support multiple working dirs in Executor | Same title as #196, and #553's body is the empty template. | | #663 Make task reset policy configurable | Same request as #189: a setting for how the scheduler recovers when an executor is lost. | | #12 Prefer assigning a job's tasks to the same executor | Covered by #2319 (shuffle affinity, open PR #2430). | | #456 Support deltalake | Covered by #1241 (open table formats). There is already an example extension at milenkovicm/ballista_delta. | ## Ask the reporter before closing - #2856 (job hangs after executors are OOM-killed): #2860 fixed one cause of the hang, found while looking into this issue, but it only closes #2859. A re-run would confirm whether anything is left. - #2849 (null-aware anti join loses its probe-side coalesce): #2848 calls this a related gap rather than fixing it, so it is probably still open. The reporter should confirm. ## Probably out of date These weren't checked against the code. They are a judgment call based on age and the APIs they refer to. - #22 refers to the old `BallistaContext`, which no longer exists. - #90 is about the old sled state backend. - #263 depends on scheduler behaviour from 2022 (#134). - #282 is a broad 2022 cleanup request. - #650 is a design discussion that stopped in 2023. ## Can be closed once an open PR merges | Issue | Open PR | |---|---| | #2882 | #2784 | | #2840 | #2867 | | #2839 | #2886 | | #2804 | #2809 | | #2767 | #2775 | | #2691 | #2698 | | #2622 | #2818 | | #2578 | #2774 | | #2454 | #2501 | | #2319 | #2430 | | #2854, #2866 | #2852 | #2371 could also be closed as won't fix, since the discussion on #2395 says the current behaviour is expected. ## Look fixed but are still open - #1770: the static planner still uses `ballista.optimizer.broadcast_join_threshold_bytes` (`planner.rs`), and #1900 was only part of this issue. - #1627: the REST API part is done (#1650), but the tree format for `EXPLAIN` is still waiting on a DataFusion protobuf change. - #2745: #2747 fixed the build with only the `rest-api` feature. This issue asks for the event-log writer to build without that feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
