erratic-pattern commented on issue #18337: URL: https://github.com/apache/datafusion/issues/18337#issuecomment-3607721424
With the error message changes in https://github.com/apache/datafusion/pull/19070 applied, the reproducer described in this issue now gives a more helpful error message: ``` Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences: - field metadata at index 1 [prev_value]: (physical) {"year": "2015"} vs (logical) {}. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker ``` It looks like the metadata from the original column is being stripped when the derived column `prev_value` is created in [`<Expr as ExprSchemable>::to_field`](https://github.com/apache/datafusion/blob/5b441a9cbec62963bf35627de6d2d54df88d3a09/datafusion/expr/src/expr_schema.rs#L457). I am not super familiar with how metadata is supposed to be handled or whether we already have library code to deal with this, but a simple hacky workaround could look something like: ```rust Expr::WindowFunction(window_function) => { let (dt, nullable) = self.data_type_and_nullable_with_window_function( schema, window_function, )?; // Preserve metadata from the first argument if it's a column let metadata = if let Some(first_arg) = window_function.params.args.first() { match first_arg.to_field(schema) { Ok((_, field)) => field.metadata().clone(), Err(_) => Default::default(), } } else { Default::default() }; let mut field = Field::new(&schema_name, dt, nullable); field.set_metadata(metadata); Ok(Arc::new(field)) } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
