alexandrefimov commented on issue #25100: URL: https://github.com/apache/datafusion/issues/25100#issuecomment-5640765255
Yes, this is also a compatibility problem. DataFusion exports a complete logical aggregate over raw input rows, but sets `phase` to `UNSPECIFIED`. Current substrait-java maps `UNSPECIFIED` to Spark `Final`, because in Substrait it means `INTERMEDIATE_TO_RESULT`. `INITIAL_TO_RESULT` maps to Spark `Complete`. As a result, substrait-java treats the aggregate arguments as intermediate state even though the plan contains expressions over the original input rows. DataFusion-to-DataFusion round trips do not catch this because the DataFusion consumer ignores `phase`. #25049 is a separate issue: substrait-java currently fails earlier because `output_type` is missing. Once that is fixed, the phase is still incorrect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
