peter-toth commented on issue #23802: [SPARK-26893][SQL] Allow pushdown of partition pruning subquery filters to file source URL: https://github.com/apache/spark/pull/23802#issuecomment-464347435 > @peter-toth I checked and the subquery is executed twice in your UT. Please set a breakpoint in `updateResult` of scalar subquery. I am not sure why honestly since in the execution plan I indeed see only one subquery exec: may you please investigate this? Moreover, it would be great if we could enforce this in a UT: the only way I can think of it to create a dummy plan on the tests which extends ScalarSubquery and keeps a global counter about the number of times updateResult is invoked. Hmm, thanks. I will look into it. Actually I just used your SPARK-25482 UT to make sure this PR doesn't cause that issue again. If this double execution gets confirmed we might need to revise that UT as well...
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
