cloud-fan commented on PR #49724: URL: https://github.com/apache/spark/pull/49724#issuecomment-2754504254
I think generating the EXPLAIN string once per query is OK, as that was the case already before AQE. The real issue is the AQE plan change event being too frequent and each event generates the EXPLAIN string once. My proposal is to only have two AQE plan change events: one for reporting the initial plan and one for reporting the final plan. We can still keep the full plan change history in the debugging logs or have a config to still generate events for them. > Additionally, when a CachedRDDBuilder object is created without a defined tableName, the full string representation of the plan is also computed, only to later extract the first 1024 characters. This sounds like a separated issue. Can we open a new PR for it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
