Jorge created ARROW-9809:
----------------------------
Summary: [Rust] [DataFusion] logical schema = physical schema is
not true
Key: ARROW-9809
URL: https://issues.apache.org/jira/browse/ARROW-9809
Project: Apache Arrow
Issue Type: Bug
Components: Rust, Rust - DataFusion
Reporter: Jorge
In tests/sql.rs, we test that the physical and the optimized schema must match.
However, this is not necessarily true for all our queries. An example:
{code:java}
#[test]
fn csv_query_sum_cast() {
let mut ctx = ExecutionContext::new();
register_aggregate_csv_by_sql(&mut ctx);
// c8 = i32; c9 = i64
let sql = "SELECT c8 + c9 FROM aggregate_test_100";
// check that the physical and logical schemas are equal
execute(&mut ctx, sql);
}
{code}
The physical expression (and schema) of this operation, after optimization, is
{{CAST(c8 as Int64) Plus c9}} (this test fails).
AFAIK, the invariant of the optimizer is that the output types and nullability
are the same.
Also, note that the reason the optimized logical schema equals the logical
schema is that our type coercer does not change the output names of the schema,
even though it re-writes logical expressions. I.e. after the optimization,
`.to_field()` of an expression may no longer match the field name nor type in
the Plan's schema. IMO this is currently by (implicit?) design, as we do not
want our logical schema's column names to change during optimizations, or all
column references may point to non-existent columns. This is something that
brought up on the mailing list about polymorphism.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
