metegenez commented on PR #1684:
URL:
https://github.com/apache/datafusion-ballista/pull/1684#issuecomment-4467341251
> thanks @metegenez i think this is good addition to AQE, it will help us
moving things, few comments and one note
>
> running TPCH SF1 Q22 i got
>
> ```
>
> thread 'tokio-rt-worker' (3376074) panicked at
ballista/scheduler/src/state/aqe/optimizer_rule/coalesce_partitions.rs:239:23:
> index out of bounds: the len is 14 but the index is 14
> note: run with `RUST_BACKTRACE=1` environment variable to display a
backtrace
> ```
I couldnt reproduce it by running
```
pkill -9 -f "ballista-(scheduler|executor)" 2>/dev/null; sleep 2
rm -rf /tmp/ballista-work && mkdir -p /tmp/ballista-work
rm -f /tmp/sch.log /tmp/exec.log
```
```
RUST_LOG="ballista=info,ballista_scheduler::state::aqe::optimizer_rule=debug" \
./target/release/ballista-scheduler --bind-port 50150 --bind-host
0.0.0.0 &>
/tmp/sch.log &
```
```
RUST_LOG=ballista=info ./target/release/ballista-executor --bind-port 50151
--bind-host 0.0.0.0 \
--scheduler-host localhost --scheduler-port 50150 \
--concurrent-tasks 8 --memory-pool-size 4GB \
--work-dir /tmp/ballista-work &> /tmp/exec.log &
```
Trials
```
./target/release/tpch benchmark ballista --host localhost --port 50150
--path benchmarks/data --format parquet --partitions 16 --iterations 1 -c
"ballista.planner.adaptive.enabled=true" -c "ballista.coalesce.enabled=true" -c
"datafusion.optimizer.prefer_hash_join=true"
```
and
```
./target/release/tpch benchmark ballista --host localhost --port 50150
--path benchmarks/data --format parquet --partitions 16 --iterations 1 -c
"ballista.planner.adaptive.enabled=true" -c "ballista.coalesce.enabled=true" -c
"datafusion.optimizer.prefer_hash_join=false"
```
What was your config on running this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
