beliefer commented on PR #42223: URL: https://github.com/apache/spark/pull/42223#issuecomment-1658176942
@peter-toth Thank you for the review. For your first point, we surely could reuse some function if there are some code is similar. If [SPARK-40193](https://issues.apache.org/jira/browse/SPARK-40193) / https://github.com/apache/spark/pull/37630. could be merged first, I will reuse them. Otherwise, this PR merged first, I will create follow up PR for reuse. For your second point, I'm OK to pass `Filter` nodes up into `Aggregate` nodes if the `tryMergePlans` can be reused. The reason about prformance degradation is `Filter` haven't better or higher selective. Please refer the benchmark in the description. So I added the config and the default value is false. Your suggestion is welcome. But I think what's the chance to reuse them? If https://github.com/apache/spark/pull/37630 merged first, I will reuse them directly. Otherwise, I will create follow up PR to fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
