jkosh44 commented on issue #16272:
URL: https://github.com/apache/datafusion/issues/16272#issuecomment-3194020152
The failure happens when converting the DF logical plan into substrait. The
logical plan of the above query looks like
```
Projection(Projection {
expr: [
Column(Column {
relation: None,
name: "column1"
})
],
input: Filter(Filter {
predicate: BinaryExpr(BinaryExpr {
left: Column(Column {
relation: None,
name: "column1"
}),
op: Eq,
right: Literal(NULL, None)
}),
input: Values(Values {
schema: DFSchema {
inner: Schema {
fields: [
Field {
name: "column1",
data_type: Int64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: {}
}
],
metadata: {}
},
field_qualifiers: [None],
functional_dependencies: FunctionalDependencies { deps: [] }
},
values: [
[Literal(Int64(1), None)]
]
})
}),
schema: DFSchema {
inner: Schema {
fields: [
Field {
name: "column1",
data_type: Int64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: {}
}
],
metadata: {}
},
field_qualifiers: [None],
functional_dependencies: FunctionalDependencies { deps: [] }
}
})
```
The conversion that fails is the right hand side of the predicate,
specifically `right: Literal(NULL, None)`. We correctly identify that the
literal is null.
https://github.com/apache/datafusion/blob/8f15991f33bf6aca9d4da8958141b59d196b2ed6/datafusion/substrait/src/logical_plan/producer/expr/literal.rs#L54-L67
However, when we see that the type of the null is `NULL`, we return an error.
https://github.com/apache/datafusion/blob/8f15991f33bf6aca9d4da8958141b59d196b2ed6/datafusion/substrait/src/logical_plan/producer/types.rs#L34-L44
I think for this to work properly, we have to know the actual type of the
null, since substrait nulls must be typed. For example if we modify the .slt to
the following
```
query I rowsort
select column1 from VALUES (1) where column1 = (NULL::INT)
----
```
it works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
