jonmmease opened a new issue, #3281:
URL: https://github.com/apache/arrow-datafusion/issues/3281
**Describe the bug**
There seems to be an interaction between the `FilterPushDown` logical
optimization and the `UNION ALL` operator that results in a `SchemaError`.
Here is the simplest example I was able to identify:
**To Reproduce**
```rust
use std::sync::Arc;
use datafusion::arrow::datatypes::Schema;
use datafusion::datasource::empty::EmptyTable;
use datafusion::prelude::SessionContext;
use vegafusion_core::arrow::datatypes::{DataType, Field};
#[tokio::test]
async fn test_query() {
let ctx = SessionContext::new();
let schema = Schema::new(vec![
Field::new("colA", DataType::Float64, true),
]);
let table = EmptyTable::new(Arc::new(schema));
ctx.register_table("tbl", Arc::new(table)).unwrap();
let df = ctx.sql(r#"
WITH
tbl_1 AS (SELECT "colA" AS "renamedA" FROM "tbl"),
tbl_2 AS ((SELECT * FROM tbl_1) UNION ALL (SELECT * FROM tbl_1))
SELECT * FROM tbl_2 WHERE ("renamedA" IS NOT NULL)
"#).await.unwrap();
let result = df.collect().await.unwrap();
}
```
```
called `Result::unwrap()` on an `Err` value: SchemaError(FieldNotFound {
qualifier: None, name: "renamedA", valid_fields: Some(["tbl.colA"]) })
```
**Expected behavior**
This code block should evaluate successfully to an empty record batch
containing the `renamedA` column.
**Additional context**
The example executes successfully if `UNION ALL ` is replaced by `UNION`.
And it executes successfully if I comment out the use of the `FilterPushDown`
in `datafusion/core/src/execution/context.rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
