tanelk edited a comment on pull request #30965: URL: https://github.com/apache/spark/pull/30965#issuecomment-752200096
A bit more information: The unstability of CBO has been noted before (#29638) and I think, that this is the main reason for this. There is also #29871, that tackles another reason for the unstability. That one should only impact, when the costs are equal, this here can impact more plans (see the example in the description) An important thing to note is that this could change the the behavior of the `spark.sql.cbo.joinReorder.card.weight` config value, but luckily it seems, that it does so minimally. I generated random values for the plan row counts and sizes, and found that the new cost function agrees most with the old cost function at the same weight value. This holds true for all the weight value, not only the default (0.7).  ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
