marvelefe opened a new pull request, #2884:
URL: https://github.com/apache/datafusion-ballista/pull/2884

   # Which issue does this PR close?
   
   Closes #2598.
   
   # Rationale for this change
   
   `execute_query_pull` and `execute_query_push` duplicated connection setup, 
job-success handling and status messages. The Substrait example had a third 
copy of the status loop. This is a refactor; pull and push are both kept, as 
agreed in the issue thread.
   
   # What changes are included in this PR?
   
   - `distributed_query.rs`: adds `connect_scheduler`, `finish_successful_job`, 
`log_job_wait` and `job_failed`. `PipelinedFetch::for_job` bundles the fetch 
settings, which removes the `too_many_arguments` allows. `fetch_partition` 
takes `PipelinedFetch`.
   - `distributed_query.rs` and `mod.rs`: adds the public 
`poll_job_until_successful` helper and exports `SchedulerClient`.
   - `standalone-substrait.rs`: replaces the private `poll_job` with the helper.
   - Tests: scripted scheduler covering success (timing gauges), failure and 
polling for pull and push.
   
   # Are there any user-facing changes?
   
   No behavior change for `DistributedQueryExec`. The new public helper is 
additive. The Substrait example now sends gRPC metadata interceptor headers 
(empty by default) and returns an error for pipelined jobs.
   
   # Verification
   
   - [x] Lint: `Lint`, `Clippy`, `Check for unused dependencies`, `Check 
Cargo.toml formatting`, `Check generated config docs are in sync`, `Check 
vendored DataFusion proto is in sync`, `cargo doc` (CI, passing)
   - [x] `distributed_query` and `ballista-core` tests: `test linux crates`, 
`macos test` (CI, passing)
   - [ ] Substrait example run (`--features ballista-scheduler/substrait`); 
compiles in CI, not run
   - [x] `ballista` client tests with `ballista.client.pull=false` (default): 
`test linux ballista` (CI, passing)
   - [ ] `ballista` client tests with `ballista.client.pull=true`; not run in 
CI or locally
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to