revans2 commented on issue #25499: [SPARK-28774] [SQL] Fix exchange reuse for columnar data URL: https://github.com/apache/spark/pull/25499#issuecomment-522713755 The rule `ReuseExchange` optimization rule will look for instances of `Exchange` that have the same plan and convert dedupe them to them to a `ReuseExchangeExec` instance. In the current Spark codebase all Exchange instances are row based, but if we use the `spark.sql.extensions` config to put in our own columnar based exchange implementation reuse will throw an exception saying that there was a columnar mismatch. I tested this patch by running it against a query that was showing this exact issue and it fixed it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
