vbarua opened a new pull request, #12245: URL: https://github.com/apache/datafusion/pull/12245
Closes https://github.com/apache/datafusion/issues/12223 ## Rationale for this change When DataFusion consumes a Substrait plan, if the schema it has a for a table differs from that given/expected by the Substrait plan it should reject the plan. Attempting to execute a plan when DataFusion and Substrait disagree on the schema is unlikely to lead to meaningful results. ## What changes are included in this PR? This PR: * Adds a validation to the the Substrait plan consumer for NamedScans that rejects Substrait plans when the schema in Substrait does not match that in DataFusion. * Updates existing tests that fail because of the above validation. * Updates the Substrait plan producer to include base schemas for ReadRels, which helps with round trip testing (this also caused test failures with the validation). * Adds an optional parameter to from_substrait_named_struct to allow the field_qualifier to be set (this also caused test failures with the validation). * Makes the from_substrait_named_struct public as it is generally useful, and also assists with testing. ## Are these changes tested? Additional tests were added in [substrait_validations.rs](https://github.com/apache/datafusion/compare/main...vbarua:datafusion:vbarua/substrait/validate-schemas?expand=1#diff-eaf82cbac0fc60b73fe8094e6dd6dbc2a0564542defa760ffdbe979f5693eac8) for the validation functionality. Existing tests failed because of the validation (correctly) and where updated to test their original functinality. ## Are there any user-facing changes? The added validation could potentially cause calls to `from_substrait_plan` that worked in prior versions to fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
