1fanwang opened a new issue, #26372: URL: https://github.com/apache/datafusion/issues/26372
### Is your feature request related to a problem or challenge? INTERSECT ALL and EXCEPT ALL are planned with an aggregate that counts the rows on each side, a join, and an unnest of range that repeats each row, as proposed in https://github.com/apache/datafusion/pull/25767. The Substrait producer has no case for the Unnest plan node, so exporting one of these plans fails with "Unsupported plan type: Unnest". The two tests at https://github.com/apache/datafusion/blob/1f95f9a0b2e058a3d561baa676fc8542492f0337/datafusion/substrait/tests/cases/roundtrip_logical_plan.rs#L1806-L1835 show the error for both operations, and the producer rejects the node at https://github.com/apache/datafusion/blob/1f95f9a0b2e058a3d561baa676fc8542492f0337/datafusion/substrait/src/logical_plan/producer/rel/mod.rs#L76. Consuming Substrait multiset set operations still works, so only the export is affected. Before that change these plans were exported as a semi join or an anti join, which returned the wrong number of duplicate rows. ### Describe the solution you'd like Support LogicalPlan::Unnest in the Substrait producer, so that INTERSECT ALL and EXCEPT ALL plans can be exported again with the correct duplicate counts. I do not know yet which Substrait relation should carry it, so I would welcome a pointer from someone who has looked at that. ### Describe alternatives you've considered None so far. ### Additional context The consuming side of multiset set operations is covered by the tests at https://github.com/apache/datafusion/blob/1f95f9a0b2e058a3d561baa676fc8542492f0337/datafusion/substrait/tests/cases/roundtrip_logical_plan.rs#L1655-L1688. ### Willingness to contribute I cannot contribute to this feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
