alamb opened a new issue, #4346: URL: https://github.com/apache/arrow-datafusion/issues/4346
**Describe the bug** Some passes may change the metadata for a plan and I think that is ok. In paticular, we have a case in IOx where the projection_push_down actually exposes some more metadata (as it unwraps a projection that had masked it). This new metadata actually fails with the newly added schema check added in https://github.com/apache/arrow-datafusion/pull/4233 by @jackwener. This currently causes our test to break **To Reproduce** Make a projection that doesn't do anything other than to mask some other plan that produces metadata ``` thread 'cases::test_cases_pushdown_sql' panicked at 'built plan successfully: Internal("Optimizer rule 'projection_push_down' failed, due to generate a different schema, original schema: DFSchema { fields: [DFField { qualifier: Some("restaurant"), field: Field { name: "count", data_type: UInt64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None } }, DFField { qualifier: Some("restaurant"), field: Field { name: "system", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None } }, DFField { qualifier: Some("restaurant"), field: Field { name: "time", data_type: Timestamp(Nanosecond, None), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None } }, DFField { qualifier: Some("restaurant"), field: Field { name: "town", data_type: Dictionary(Int32, Utf8), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None } }], metadata: {} }, new schema: DFSchema { fields: [DFField { qualifier: Some("restaurant"), field: Fiel d { name: "count", data_type: UInt64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: Some({"iox::column::type": "iox::column_type::field::uinteger"}) } }, DFField { qualifier: Some("restaurant"), field: Field { name: "system", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: Some({"iox::column::type": "iox::column_type::field::float"}) } }, DFField { qualifier: Some("restaurant"), field: Field { name: "time", data_type: Timestamp(Nanosecond, None), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: Some({"iox::column::type": "iox::column_type::timestamp"}) } }, DFField { qualifier: Some("restaurant"), field: Field { name: "town", data_type: Dictionary(Int32, Utf8), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: Some({"iox::column::type": "iox::column_type::tag"}) } }], metadata: {} }")', query_tests/src/runner.rs:289:18 ``` **Expected behavior** The query should run successfully **Additional context** I also considered adding a config flag to turn off the check, but that seems worse than this less stringent test Here is the PR where i am working to upgrade datafusion: https://github.com/influxdata/influxdb_iox/pull/6218#issuecomment-1325323048 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
