mmooyyii opened a new issue, #1235:
URL: https://github.com/apache/datafusion-ballista/issues/1235
**Describe the bug**
run example custom-scheduler in push-staged mode, it returns
```
Error: ArrowError(ExternalError(Execution("Job ScIzlxB failed: Job failed
due to stage 1 failed: Task failed due to runtime execution error:
DataFusionError(Execution(\"DataFusionError(Configuration(\\\"'s3.access_key_id'
& 's3.secret_access_key' must be configured\\\"))\"))\n")), None)
```
**To Reproduce**
use custom-client.rs, custom-executor.rs,custom-scheduler.rs in example
1. change the default TaskSchedulingPolicy to push staged in
datafusion-ballista/ballista/core/src/config.rs
```rust
pub enum TaskSchedulingPolicy {
/// push-based scheduling can result in lower latency.
#[default]
PushStaged,
/// Pull-based scheduling works in a similar way to Apache Spark
PullStaged,
}
```
2. run custom-scheduler, custom-executor and custom-client
```
yimo@YideMacBook-Pro examples % cargo run --example custom-scheduler
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
Running
`/Users/yimo/RustroverProjects/datafusion-ballista/target/debug/examples/custom-scheduler`
[2025-04-11T06:05:34Z INFO ballista_scheduler::scheduler_process] Ballista
v45.0.0 Scheduler listening on 127.0.0.1:50050
[2025-04-11T06:05:34Z INFO ballista_scheduler::scheduler_process] Starting
Scheduler grpc server with task scheduling policy of PushStaged
[2025-04-11T06:05:34Z INFO
ballista_scheduler::scheduler_server::query_stage_scheduler] Starting
QueryStageScheduler
[2025-04-11T06:05:34Z INFO ballista_core::event_loop] Starting the event
loop query_stage
```
```
yimo@YideMacBook-Pro datafusion-ballista % cargo run --example
custom-executor
Compiling ballista-examples v45.0.0
(/Users/yimo/RustroverProjects/datafusion-ballista/examples)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.47s
Running `target/debug/examples/custom-executor`
[2025-04-11T06:05:53Z INFO ballista_executor::executor_process] Executor
starting ... (Datafusion Ballista 45.0.0)
[2025-04-11T06:05:53Z INFO ballista_executor::executor_process] Executor
working directory: /var/folders/yz/sy3r3gjs22j3bv_vt58jnzw00000gn/T/.tmpc2qJpH
[2025-04-11T06:05:53Z INFO ballista_executor::executor_process] Executor
number of concurrent tasks: 10
[2025-04-11T06:05:53Z INFO ballista_executor::executor_server] Ballista
v45.0.0 Rust Executor Grpc Server listening on 127.0.0.1:50052
[2025-04-11T06:05:53Z INFO ballista_executor::executor_server] Executor
registration succeed
[2025-04-11T06:05:53Z INFO ballista_executor::executor_process] Starting
built-in arrow flight service
[2025-04-11T06:05:53Z INFO ballista_executor::executor_server] Starting
heartbeater to send heartbeat the scheduler periodically
[2025-04-11T06:05:53Z INFO ballista_executor::executor_server] Starting the
task status reporter
[2025-04-11T06:05:53Z INFO ballista_executor::executor_server] Starting the
task runner pool
[2025-04-11T06:05:53Z INFO ballista_executor::executor_process] Built-in
arrow flight server listening on: 127.0.0.1:50051 max_encoding_size: 16777216
max_decoding_size: 16777216
```
```
yimo@YideMacBook-Pro datafusion-ballista % cargo run --example custom-client
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.17s
Running `target/debug/examples/custom-client`
++
++
++
++
++
++
++
++
++
++
Error: ArrowError(ExternalError(Execution("Job ScIzlxB failed: Job failed
due to stage 1 failed: Task failed due to runtime execution error:
DataFusionError(Execution(\"DataFusionError(Configuration(\\\"'s3.access_key_id'
& 's3.secret_access_key' must be configured\\\"))\"))\n")), None)
```
BTW. object store example can't use functions like count or avg;
It should add .with_default_features() in config;
```
// datafusion-ballista/examples/src/object_store.rs
pub fn custom_session_state_with_s3_support(
session_config: SessionConfig,
) -> datafusion::common::Result<SessionState> {
let runtime_env = custom_runtime_env_with_s3_support(&session_config)?;
Ok(SessionStateBuilder::new()
.with_runtime_env(runtime_env)
.with_config(session_config)
.build())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
