2010YOUY01 commented on issue #23461: URL: https://github.com/apache/datafusion/issues/23461#issuecomment-5082515443
Should we keep the error behavior, but provide a better error message? 🤔 From the Arrow spec, after joining two tables with conflicting metadata key, what's the expected behavior is unspecified, so I tend to think we should be more conservative. https://arrow.apache.org/docs/format/Columnar.html#schema-message After the fix in [https://github.com/apache/datafusion/pull/23605](https://github.com/apache/datafusion/pull/23605), the behavior becomes deterministic: when a metadata key conflicts, the value from the original right child of `CrossJoinExec` is used. Silently allowing one side to override the other is fairly confusing. It could propagate inconsistent metadata and trigger another bug later. However if certain application prefer such overriding behavior, we could specify it first (somewhere in arrow schema metadata spec), and then continue the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
