SteveLauC commented on issue #9405:
URL:
https://github.com/apache/arrow-datafusion/issues/9405#issuecomment-1974114287
> It seems it is because of this in your code:
Can confirm that commenting this function will make the test pass.
-----
However I am still surprised at DataFusion's behavior:
The following is the generated physical plan:
```rs
CoalesceBatchesExec {
input: FilterExec {
predicate: BinaryExpr {
left: Column {
name: "id",
index: 0,
},
op: Gt,
right: Literal {
value: Int32(1),
},
},
input: RepartitionExec {
input: TestExecutionPlan {
schema: Schema {
fields: [
Field {
name: "id",
data_type: Int32,
nullable: false,
dict_id: 0,
dict_is_ordered: false,
metadata: {},
},
Field {
name: "title",
data_type: Utf8,
nullable: false,
dict_id: 0,
dict_is_ordered: false,
metadata: {},
},
],
metadata: {},
},
},
partitioning: RoundRobinBatch(
32,
),
state: Mutex {
data: RepartitionExecState {
channels: {},
abort_helper: AbortOnDropMany(
[],
),
},
},
metrics: ExecutionPlanMetricsSet {
inner: Mutex {
data: MetricsSet {
metrics: [],
},
},
},
preserve_order: false,
},
metrics: ExecutionPlanMetricsSet {
inner: Mutex {
data: MetricsSet {
metrics: [],
},
},
},
default_selectivity: 20,
},
target_batch_size: 8192,
metrics: ExecutionPlanMetricsSet {
inner: Mutex {
data: MetricsSet {
metrics: [],
},
},
},
}
```
We can see that the first filter cannot be pushed down so there is a
`FilterExec` for it, but the second one, DataFusion didn't put it in the
`FilterExec`, and didn't pushed it down:
```sh
TestTableProvider::scan() filters: []
```
That incorrect implementation of
`TableProvider::supports_filters_pushdown()` seems to be an edge case that
hasn't be covered by the code :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
