marvelefe opened a new pull request, #2884: URL: https://github.com/apache/datafusion-ballista/pull/2884
# Which issue does this PR close? Closes #2598. # Rationale for this change `execute_query_pull` and `execute_query_push` duplicated connection setup, job-success handling and status messages. The Substrait example had a third copy of the status loop. This is a refactor; pull and push are both kept, as agreed in the issue thread. # What changes are included in this PR? - `distributed_query.rs`: adds `connect_scheduler`, `finish_successful_job`, `log_job_wait` and `job_failed`. `PipelinedFetch::for_job` bundles the fetch settings, which removes the `too_many_arguments` allows. `fetch_partition` takes `PipelinedFetch`. - `distributed_query.rs` and `mod.rs`: adds the public `poll_job_until_successful` helper and exports `SchedulerClient`. - `standalone-substrait.rs`: replaces the private `poll_job` with the helper. - Tests: scripted scheduler covering success (timing gauges), failure and polling for pull and push. # Are there any user-facing changes? No behavior change for `DistributedQueryExec`. The new public helper is additive. The Substrait example now sends gRPC metadata interceptor headers (empty by default) and returns an error for pipelined jobs. # Verification - [x] Lint: `Lint`, `Clippy`, `Check for unused dependencies`, `Check Cargo.toml formatting`, `Check generated config docs are in sync`, `Check vendored DataFusion proto is in sync`, `cargo doc` (CI, passing) - [x] `distributed_query` and `ballista-core` tests: `test linux crates`, `macos test` (CI, passing) - [ ] Substrait example run (`--features ballista-scheduler/substrait`); compiles in CI, not run - [x] `ballista` client tests with `ballista.client.pull=false` (default): `test linux ballista` (CI, passing) - [ ] `ballista` client tests with `ballista.client.pull=true`; not run in CI or locally -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
