This is an automated email from the ASF dual-hosted git repository.
martin-g pushed a change to branch redirect-root-to-nightlies
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
omit a820cc95b Use X-Forwarded-XYZ headers to read the protocol+host+port
omit b342c1fa5 clippy
omit aed9569fd Use config's external_host & bind_port as a fallback for the
redirect url to WebTUI
omit 327979a09 Make it possible to use custom Axum route for the path
redirecting to the WebTUI at https://nightlies.apache.org
omit 663887990 Merge branch 'main' into redirect-root-to-nightlies
omit 1c81a45fa Re-enable the CI condition for deployment at nightlies.a.o
omit fce4b6390 Update python/Cargo.lock
omit a7900b468 Own the parsed query parameter values
omit b0d905631 Import std::collections::HashMap
omit d99f33fa4 Use "See Other" redirect (code 303)
omit e5dabccca URL encode the query parameters
omit a9f4b9fa7 Re-enable the CI job condition to deploy WebTUI only on push
to main
omit 467b4e5a8 Fix rsync delete command: `--delete does not work without
--recursive or --dirs`
omit f68f13801 feat(scheduler): Redirect '/' to nighlies.a.o WebTUI
add fd5aa65d6 feat: make broadcast_join_threshold_bytes/rows authoritative
under AQE and the static planner (#2086)
add 847ee7c45 feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes
(#2088)
add 2ed3464df Multi-partition tasks: partition_slice plumbing +
SortShuffleWriter refactor (#2038)
add 968086f6c docs(config): rewrite max_partitions_per_task description to
be timeless (#2093)
add 752d4de13 fix: propagate sort shuffle memory_limit_per_task_bytes to
executors (#2090)
add 5408c7613 feat: allow an external semaphore for executor task
concurrency (#1892)
add 2d938371a feat: log sort shuffle spills at WARN, keep write completion
at DEBUG (#2091)
add 9ca2d354b chore(deps): bump async-trait from 0.1.89 to 0.1.91 (#2100)
add 0d7253d71 chore(deps): bump futures from 0.3.32 to 0.3.33 (#2099)
add 92198ed12 chore(deps): bump serde from 1.0.228 to 1.0.229 (#2097)
add f3542e00f chore(ci): bump taiki-e/install-action from 2.83.3 to 2.83.4
(#2096)
add 49d1fec80 chore(ci): bump actions/setup-python from 6 to 7 (#2098)
add 60c5b26c9 chore(scheduler,executor): downgrade #2038 per-task traces
to debug (#2104)
add d518d569a feat(benchmarks): persist TPC-H results resiliently and add
a compare subcommand (#2102)
add c77e48127 Fixed: notify_shutdown sending signal before drop (#2107)
add 99ba9e860 docs: refresh SF1000 benchmark results on 2x16-core AQE-on
reference cluster (#2109)
add 0e0a695e1 chore(deps): bump clap from 4.6.2 to 4.6.3 (#2118)
add f8efcdf9a chore(deps): bump libc from 0.2.186 to 0.2.187 (#2116)
add 826834a09 chore(deps): bump tokio from 1.53.0 to 1.53.1 in /python
(#2115)
add 97dab80e1 chore(deps): bump tokio from 1.53.0 to 1.53.1 (#2114)
add 81a184ecf chore(ci): bump actions/checkout from 7.0.0 to 7.0.1 (#2113)
add e6f657569 chore(ci): bump actions/labeler from 6.2.0 to 7.0.0 (#2112)
add 0dac3493d chore(deps): bump serde_json from 1.0.150 to 1.0.151 (#2111)
add d3a8724e9 chore(ci): bump taiki-e/install-action from 2.83.4 to 2.84.0
(#2110)
add f6376add6 feat(scheduler): Redirect '/' to nighlies.a.o WebTUI
add ad9525ded Fix rsync delete command: `--delete does not work without
--recursive or --dirs`
add 1cdb88a26 Re-enable the CI job condition to deploy WebTUI only on push
to main
add cde3f5961 URL encode the query parameters
add c6b255f4f Use "See Other" redirect (code 303)
add f3cabe251 Import std::collections::HashMap
add 54a34e6d6 Own the parsed query parameter values
add 824699fc7 Update python/Cargo.lock
add 97ea732d8 Re-enable the CI condition for deployment at nightlies.a.o
add 8d9175e44 Make it possible to use custom Axum route for the path
redirecting to the WebTUI at https://nightlies.apache.org
add 3371f6352 Use config's external_host & bind_port as a fallback for the
redirect url to WebTUI
add b9077055b clippy
add 429cac677 Use X-Forwarded-XYZ headers to read the protocol+host+port
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a820cc95b)
\
N -- N -- N refs/heads/redirect-root-to-nightlies (429cac677)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/allowlist-check.yml | 2 +-
.github/workflows/build.yml | 20 +-
.github/workflows/codeql.yml | 2 +-
.github/workflows/dependencies.yml | 4 +-
.github/workflows/dev.yml | 6 +-
.github/workflows/dev_pr.yml | 2 +-
.github/workflows/docker.yml | 2 +-
.github/workflows/docs.yaml | 6 +-
.github/workflows/rust.yml | 26 +-
.github/workflows/tpcds.yml | 2 +-
.github/workflows/tpch.yml | 4 +-
.github/workflows/web-tui.yml | 6 +-
.gitignore | 1 +
Cargo.lock | 126 +--
ballista/core/proto/ballista.proto | 60 +-
ballista/core/src/config.rs | 57 +-
.../execution_plans/distributed_explain_analyze.rs | 3 +
ballista/core/src/execution_plans/mod.rs | 1 +
.../core/src/execution_plans/shuffle_reader.rs | 5 +-
.../core/src/execution_plans/shuffle_writer.rs | 1080 +++++++++++++++-----
.../src/execution_plans/sort_shuffle/buffer.rs | 5 +-
.../src/execution_plans/sort_shuffle/config.rs | 4 +-
.../src/execution_plans/sort_shuffle/writer.rs | 535 ++++++++--
ballista/core/src/extension.rs | 52 +-
ballista/core/src/lib.rs | 12 +
ballista/core/src/serde/generated/ballista.rs | 74 +-
ballista/core/src/serde/mod.rs | 56 +-
ballista/core/src/serde/scheduler/from_proto.rs | 73 +-
ballista/core/src/serde/scheduler/mod.rs | 35 +-
ballista/core/src/serde/scheduler/to_proto.rs | 132 ++-
ballista/executor/Cargo.toml | 3 +-
ballista/executor/src/bin/main.rs | 27 +-
ballista/executor/src/config.rs | 8 +
ballista/executor/src/execution_engine.rs | 463 ++++-----
ballista/executor/src/execution_loop.rs | 93 +-
ballista/executor/src/executor.rs | 81 +-
ballista/executor/src/executor_process.rs | 89 +-
ballista/executor/src/executor_server.rs | 84 +-
ballista/executor/src/health.rs | 143 +++
ballista/executor/src/lib.rs | 23 +-
ballista/executor/src/metrics/mod.rs | 6 +-
ballista/executor/src/runtime_cache.rs | 51 +-
ballista/executor/src/standalone.rs | 11 +-
ballista/scheduler/src/api/handlers.rs | 191 ++--
ballista/scheduler/src/api/health.rs | 163 +++
ballista/scheduler/src/api/mod.rs | 2 +
ballista/scheduler/src/cluster/mod.rs | 382 ++++---
ballista/scheduler/src/config.rs | 11 +
ballista/scheduler/src/metrics/mod.rs | 22 +
ballista/scheduler/src/planner.rs | 176 +++-
ballista/scheduler/src/scheduler_process.rs | 11 +-
ballista/scheduler/src/scheduler_server/grpc.rs | 182 +++-
ballista/scheduler/src/scheduler_server/mod.rs | 32 +-
.../src/scheduler_server/query_stage_scheduler.rs | 12 +-
.../src/state/aqe/execution_plan/dynamic_join.rs | 197 +++-
ballista/scheduler/src/state/aqe/mod.rs | 171 ++--
.../scheduler/src/state/aqe/test/job_failure.rs | 20 +-
.../scheduler/src/state/aqe/test/join_selection.rs | 5 +
ballista/scheduler/src/state/execution_graph.rs | 310 +++---
ballista/scheduler/src/state/execution_stage.rs | 891 +++++++++++-----
ballista/scheduler/src/state/executor_manager.rs | 1 -
ballista/scheduler/src/state/mod.rs | 4 +-
ballista/scheduler/src/state/task_builder.rs | 591 +++++++++++
ballista/scheduler/src/state/task_manager.rs | 471 +++------
ballista/scheduler/src/test_utils.rs | 47 +-
benchmarks/README.md | 22 +
benchmarks/src/bin/tpch.rs | 629 +++++++++---
docs/developer/architecture.md | 42 +
docs/source/contributors-guide/benchmarking.md | 176 ++--
docs/source/user-guide/configs.md | 18 +-
docs/source/user-guide/deployment/kubernetes.md | 98 ++
docs/source/user-guide/tuning-guide.md | 60 +-
examples/examples/mtls-cluster.rs | 5 +-
python/Cargo.lock | 713 +++++--------
74 files changed, 6250 insertions(+), 2880 deletions(-)
create mode 100644 ballista/executor/src/health.rs
create mode 100644 ballista/scheduler/src/api/health.rs
create mode 100644 ballista/scheduler/src/state/task_builder.rs
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
