liuzqt commented on PR #40812: URL: https://github.com/apache/spark/pull/40812#issuecomment-1967661413
I'm revisiting this PR since it introduced a new issue https://issues.apache.org/jira/browse/SPARK-47177 I don't have a clear idea yet how to fix this, a few ideas out of my head: - maybe we just a coarse granularity lock in explain? - alternatively, make `innerChildren` a function: clone the initial plan, every time check for whether the original AQE plan is finalized (making the final flag atomic first, of course), if no: return the cloned initial plan, if it's finalized: clone the final plan and return that one. But still this won't be able to reflect the AQE plan in real time, in a concurrent situation, but at least we have initial version and final version. cc @cloud-fan to provide some feedback -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
