This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a change to branch chore/track-datafusion-git-main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
from b1b5f48da test: regenerate TPC-H plan-stability golden files for
DataFusion main
add 67ae6263f chore(ci): bump taiki-e/install-action from 2.84.0 to 2.85.0
(#2164)
add d9e189b3e chore(deps): bump ctor from 1.0.9 to 1.0.10 (#2137)
add 561037c7b ci: allow the docs site deploy to be triggered manually
(#2177)
add a9bf1168c chore(deps): bump libc from 0.2.187 to 0.2.189 (#2142)
add 9af0a28ca chore(deps): bump tokio-util from 0.7.18 to 0.7.19 (#2135)
add 5e6201e96 feat(core): add OrderedRangeRepartitionExec — N sorted
overlapping → K sorted disjoint (#2169)
add b9711402b kll: add native Rust KLL quantile sketch (#2180)
add be65da313 chore(deps): bump arrow from 58.3.0 to 58.4.0 (#2178)
add 1517c94f2 docs: add 54.0.0 changelog (#2012)
add bdc21e641 chore(deps): bump quinn-proto from 0.11.14 to 0.11.16 (#2176)
add 5dc768b68 fix: unskip five TPC-DS queries by rewriting generator
column names (#2183)
add 8b3ec30c8 chore(ci): bump taiki-e/install-action from 2.85.0 to 2.85.2
(#2191)
add 0deac135d chore(deps): bump aws-config from 1.9.0 to 1.10.1 (#2190)
add 99897a051 chore(deps): bump ctor from 1.0.10 to 1.0.11 (#2189)
add da96bff15 chore(ci): bump the github-codeql group across 1 directory
with 2 updates (#2133)
add 33d9bac62 chore(ci): bump astral-sh/setup-uv from 8.3.2 to 9.0.0
(#2134)
add dc6b38097 feat(core,scheduler): transport RuntimeStatsExec reports to
scheduler; log merged quantile cuts per stage (#2175)
add 828d6f007 fix: stop every task running every union branch at low
partition counts (#2186)
add ba46698f3 slice 1b3a: PerPartitionFilterExec operator + codec (#2195)
add 2b04f83ad ci: run TPC-DS SF1 with 1 and 4 partitions per task (#2197)
add bd5563ade fix(scheduler): execute null-aware anti joins in one task
(#2188)
add 7eeff3beb fix(scheduler): keep GROUP BY-less aggregates when
propagating empty stages (#2194)
add 6bbcc077a chore: use taplo for Cargo.toml formatting (#2179)
add 86fc9a766 fix(scheduler): harden PropagateEmptyExecRule against
grouping sets, embedded projections, and dropped exchange partitioning (#2200)
add d32877b85 feat(core): rewrite NOT IN subqueries into distributable
anti join plans (#2199)
add b34144e5e test: add HA chaos harness that drives the scheduler's
fault-tolerance paths on a real cluster (#2026)
add 0c7b707f4 chore(ci): bump taiki-e/install-action from 2.85.2 to 2.85.3
(#2203)
add 03fc4be2a chore(deps): bump rustls from 0.23.42 to 0.23.43 (#2209)
add 234936485 chore(deps): bump http from 1.4.2 to 1.5.0 (#2208)
add 1bbd47725 chore(deps): bump ctor from 1.0.11 to 1.0.12 (#2206)
add 017386fcc fix(cli): sort jobs by stage completion ratio (#1947)
add 39092aea1 chore(ci): bump taiki-e/install-action from 2.85.3 to 2.85.4
(#2205)
add bb9757fd4 chore(deps): bump nix from 0.29.0 to 0.31.3 (#2207)
add 187a65063 chore(deps): bump reqwest from 0.12.28 to 0.13.4 (#2210)
add 6e9ff68a5 fix(scheduler): preserve stages on transition errors (#2215)
add c43e56eb4 feat(scheduler): range-repartition rule primitives
(batteries-included) (#2196)
add 63fa496b5 chore(ci): bump the github-codeql group with 2 updates
(#2218)
add a3161a301 fix(scheduler): fail running jobs when all executors are
lost (#2029) (#2212)
add 9fecadae8 feat: make gRPC max message size consistently configurable
(#2173)
add 984513ee7 fix(executor): bind the gRPC listener before registering
with the scheduler (#2225)
add 10d4d0877 feat: add `poll_now_notify` to `poll_loop` and
`on_work_available` callback (#1893)
add 5355a36f4 fix: update mTLS example for poll loop arguments (#2229)
add 0df207e13 chore(deps): bump bytesize from 2.4.2 to 2.7.0 (#2227)
add 947f7d901 chore(ci): bump taiki-e/install-action from 2.85.4 to 2.85.7
(#2228)
add 121f136bf chore(deps): bump cryptography from 48.0.1 to 50.0.0 in
/python (#2221)
add f1bdc9f3b feat(scheduler): expose existing job state event stream
(#1891)
add e0c4032e2 feat(scheduler): Redirect '/' to nighlies.a.o WebTUI (#2059)
add 28c26eabe ci: add h2o window suite CI job (#2230)
add 660665d7b chore(ci): bump astral-sh/setup-uv from 8.2.0 to 9.0.0
(#2232)
add cc3c0e95b chore(deps): bump pyo3-build-config from 0.29.0 to 0.29.1 in
/python (#2233)
add fc1655d28 chore(ci): bump the github-codeql group with 2 updates
(#2231)
add 29317fb5d Merge apache/main into chore/track-datafusion-git-main
add bd6b63ce5 chore(deps): bump DataFusion rev to 70c26a06 and port to its
APIs
No new revisions were added by this update.
Summary of changes:
.github/workflows/build.yml | 14 +-
.github/workflows/codeql.yml | 4 +-
.github/workflows/docs.yaml | 3 +
.github/workflows/{tpcds.yml => h2o.yml} | 81 +-
.github/workflows/rust.yml | 16 +-
.github/workflows/tpcds.yml | 73 +-
.github/workflows/tpch.yml | 2 +-
.github/workflows/web-tui.yml | 8 +-
Cargo.lock | 154 ++-
Cargo.toml | 44 +-
ballista-cli/Cargo.toml | 64 +-
ballista-cli/src/tui/domain/jobs.rs | 41 +-
ballista-cli/src/tui/http_client.rs | 3 +-
ballista-cli/src/tui/infrastructure/config.rs | 15 +-
ballista/client/Cargo.toml | 6 +-
ballista/client/tests/context_setup.rs | 7 +-
ballista/client/tests/null_aware.rs | 152 +++
ballista/core/proto/ballista.proto | 60 +
ballista/core/proto/datafusion_common.proto | 2 +
ballista/core/src/config.rs | 60 +-
ballista/core/src/execution_plans/mod.rs | 13 +-
.../execution_plans/ordered_range_repartition.rs | 931 ++++++++++++++++
.../src/execution_plans/per_partition_filter.rs | 571 ++++++++++
ballista/core/src/execution_plans/plan_algebra.rs | 70 ++
.../execution_plans/range_repartition_common.rs | 82 +-
ballista/core/src/execution_plans/runtime_stats.rs | 1166 +++++++++++++++++++-
.../core/src/execution_plans/shuffle_writer.rs | 48 +-
ballista/core/src/extension.rs | 5 +
ballista/core/src/kll.rs | 670 +++++++++++
ballista/core/src/lib.rs | 4 +
ballista/core/src/optimizer.rs | 475 ++++++++
ballista/core/src/planner.rs | 5 +-
ballista/core/src/serde/generated/ballista.rs | 80 +-
ballista/core/src/serde/mod.rs | 215 +++-
ballista/core/src/utils.rs | 60 +
ballista/executor/Cargo.toml | 10 +-
ballista/executor/src/config.rs | 4 +-
ballista/executor/src/execution_engine.rs | 37 +
ballista/executor/src/execution_loop.rs | 40 +-
ballista/executor/src/executor_process.rs | 1 +
ballista/executor/src/executor_server.rs | 24 +-
ballista/executor/src/lib.rs | 32 +-
ballista/executor/src/standalone.rs | 1 +
ballista/scheduler/Cargo.toml | 4 +-
ballista/scheduler/src/api/handlers.rs | 297 ++++-
ballista/scheduler/src/api/routes.rs | 3 +
ballista/scheduler/src/cluster/memory.rs | 66 +-
ballista/scheduler/src/config.rs | 125 ++-
.../src/physical_optimizer/join_selection.rs | 130 ++-
ballista/scheduler/src/planner.rs | 483 ++++++--
ballista/scheduler/src/scheduler_server/grpc.rs | 2 +-
ballista/scheduler/src/scheduler_server/mod.rs | 41 +-
.../src/scheduler_server/query_stage_scheduler.rs | 419 ++++++-
ballista/scheduler/src/state/aqe/adapter.rs | 24 +-
.../src/state/aqe/execution_plan/dynamic_join.rs | 56 +-
.../src/state/aqe/execution_plan/exchange.rs | 147 +++
ballista/scheduler/src/state/aqe/mod.rs | 123 ++-
.../aqe/optimizer_rule/coalesce_partitions.rs | 10 +
.../aqe/optimizer_rule/distributed_exchange.rs | 409 ++++++-
.../src/state/aqe/optimizer_rule/join_selection.rs | 152 ++-
.../state/aqe/optimizer_rule/propagate_empty.rs | 287 ++++-
ballista/scheduler/src/state/aqe/planner.rs | 60 +-
.../scheduler/src/state/aqe/test/coalesce_rule.rs | 61 +
ballista/scheduler/src/state/aqe/test/mod.rs | 3 +
.../src/state/aqe/test/range_repartition.rs | 254 +++++
ballista/scheduler/src/state/execution_graph.rs | 227 +++-
ballista/scheduler/src/state/execution_stage.rs | 139 ++-
ballista/scheduler/src/state/executor_manager.rs | 72 +-
ballista/scheduler/src/state/task_builder.rs | 204 +++-
ballista/scheduler/src/state/task_manager.rs | 51 +-
ballista/scheduler/src/test_utils.rs | 62 +-
.../rust_toml_fmt.sh => benchmarks/pyproject.toml | 10 +-
benchmarks/queries/h2o/groupby.sql | 19 +
benchmarks/queries/h2o/join.sql | 9 +
benchmarks/queries/h2o/window.sql | 150 +++
benchmarks/src/bin/h2o.rs | 540 +++++++++
benchmarks/src/bin/tpcds.rs | 123 ++-
chaos-testing/Cargo.toml | 55 +
chaos-testing/README.md | 351 ++++++
chaos-testing/src/bin/chaos-executor.rs | 62 ++
chaos-testing/src/bin/chaos-scheduler.rs | 74 ++
chaos-testing/src/budget.rs | 126 +++
chaos-testing/src/cluster.rs | 764 +++++++++++++
chaos-testing/src/fixture.rs | 373 +++++++
.../mod.rs => chaos-testing/src/lib.rs | 18 +-
chaos-testing/src/registry.rs | 103 ++
chaos-testing/src/udf.rs | 328 ++++++
chaos-testing/tests/common/mod.rs | 131 +++
chaos-testing/tests/ha.rs | 418 +++++++
ci/scripts/rust_toml_fmt.sh | 23 +-
dev/rust_lint.sh | 6 +-
docs/source/changelog/54.0.0.md | 150 +++
docs/source/changelog/index.md | 1 +
docs/source/upgrading/55.0.0.md | 44 +
docs/source/user-guide/configs.md | 3 +-
docs/source/user-guide/tuning-guide.md | 17 +-
examples/Cargo.toml | 2 +-
examples/examples/mtls-cluster.rs | 2 +-
python/uv.lock | 99 +-
taplo.toml | 63 ++
100 files changed, 12697 insertions(+), 631 deletions(-)
copy .github/workflows/{tpcds.yml => h2o.yml} (70%)
create mode 100644 ballista/client/tests/null_aware.rs
create mode 100644
ballista/core/src/execution_plans/ordered_range_repartition.rs
create mode 100644 ballista/core/src/execution_plans/per_partition_filter.rs
create mode 100644 ballista/core/src/execution_plans/plan_algebra.rs
create mode 100644 ballista/core/src/kll.rs
create mode 100644 ballista/core/src/optimizer.rs
create mode 100644 ballista/scheduler/src/state/aqe/test/range_repartition.rs
copy ci/scripts/rust_toml_fmt.sh => benchmarks/pyproject.toml (80%)
mode change 100755 => 100644
create mode 100644 benchmarks/queries/h2o/groupby.sql
create mode 100644 benchmarks/queries/h2o/join.sql
create mode 100644 benchmarks/queries/h2o/window.sql
create mode 100644 benchmarks/src/bin/h2o.rs
create mode 100644 chaos-testing/Cargo.toml
create mode 100644 chaos-testing/README.md
create mode 100644 chaos-testing/src/bin/chaos-executor.rs
create mode 100644 chaos-testing/src/bin/chaos-scheduler.rs
create mode 100644 chaos-testing/src/budget.rs
create mode 100644 chaos-testing/src/cluster.rs
create mode 100644 chaos-testing/src/fixture.rs
copy ballista/scheduler/src/state/aqe/optimizer_rule/mod.rs =>
chaos-testing/src/lib.rs (75%)
create mode 100644 chaos-testing/src/registry.rs
create mode 100644 chaos-testing/src/udf.rs
create mode 100644 chaos-testing/tests/common/mod.rs
create mode 100644 chaos-testing/tests/ha.rs
create mode 100644 docs/source/changelog/54.0.0.md
create mode 100644 taplo.toml
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
