ygf11 opened a new issue, #5022:
URL: https://github.com/apache/arrow-datafusion/issues/5022
**Describe the bug**
A clear and concise description of what the bug is.
When I work on #4866, I find some `NestedLoopJoin`s will panic.
```sql
// create tables -- t.csv is an empty file.
❯ CREATE EXTERNAL TABLE t1 (t1_id INT, t1_name text, t1_int INT) STORED AS
CSV LOCATION 't.csv';
❯ CREATE EXTERNAL TABLE t2 (t2_id INT, t2_name text, t2_int INT) STORED AS
CSV LOCATION 't.csv';
// target_partitions
> set datafusion.execution.target_partitions = 4;
```
```sql
❯ explain select * from t1 inner join t2 on t1.t1_id > t2.t2_id where
t1.t1_id > 10 and t2.t2_int > 10;
+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Projection: t1.t1_id, t1.t1_name, t1.t1_int, t2.t2_id,
t2.t2_name, t2.t2_int
|
| | Inner Join: Filter: t1.t1_id > t2.t2_id
|
| | Filter: t1.t1_id > Int32(10)
|
| | TableScan: t1 projection=[t1_id, t1_name, t1_int],
partial_filters=[t1.t1_id > Int32(10)]
|
| | Filter: t2.t2_int > Int32(10)
|
| | TableScan: t2 projection=[t2_id, t2_name, t2_int],
partial_filters=[t2.t2_int > Int32(10)]
|
| physical_plan | ProjectionExec: expr=[t1_id@0 as t1_id, t1_name@1 as
t1_name, t1_int@2 as t1_int, t2_id@3 as t2_id, t2_name@4 as t2_name, t2_int@5
as t2_int] |
| | NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr {
left: Column { name: "t1_id", index: 0 }, op: Gt, right: Column { name:
"t2_id", index: 1 } } |
| | CoalesceBatchesExec: target_batch_size=8192
|
| | FilterExec: t1_id@0 > 10
|
| | RepartitionExec: partitioning=RoundRobinBatch(4),
input_partitions=1
|
| | CsvExec: files={1 group:
[[home/work/tools/datafusion-test-data/join-context/t1.csv]]},
has_header=false, limit=None, projection=[t1_id, t1_name, t1_int] |
| | CoalescePartitionsExec
|
| | CoalesceBatchesExec: target_batch_size=8192
|
| | FilterExec: t2_int@2 > 10
|
| | RepartitionExec:
partitioning=RoundRobinBatch(4), input_partitions=1 -- panic in RepartitionExec
|
| | CsvExec: files={1 group:
[[home/work/tools/datafusion-test-data/join-context/t1.csv]]},
has_header=false, limit=None, projection=[t2_id, t2_name, t2_int] |
| |
|
+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
This sql will panic:
```sql
❯ select * from t1 inner join t2 on t1.t1_id > t2.t2_id where t1.t1_id > 10
and t2.t2_int > 10;
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
thread 'tokio-runtime-worker' panicked at 'partition not used yet',
/home/work/arrow-datafusion/datafusion/core/src/physical_plan/repartition/mod.rs:426:14
0 rows in set. Query took 0.015 seconds.
```
**To Reproduce**
See above.
**Expected behavior**
The join should work.
It seems each partition of the right child may execute one more time in
`NestedLoopJoin`, if it has multiple output partitions. It is ok in most time,
but when meets `RepartitionExec` it will panic.
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
