tanelk commented on pull request #30965: URL: https://github.com/apache/spark/pull/30965#issuecomment-752200096
A bit more information: The unstability of CBO has been noted before (#29638) and I think, that this is the main reason for this. There is also #29871, that tackles another reason for the unstability. That one should only impact, when the costs are equal, this here can impact more plans (see the example in the description) I'm aware that the UTs will fail because the plans will be different. Because it would be quite a lot of work, I will fix those iff I get some feedback, that this issue is worth looking at further. An important thing to note is that this could change the the behavior of the `spark.sql.cbo.joinReorder.card.weight` config value, but luckily it seems, that it does so minimally. I generated random values for the plan row counts and sizes, and found that the new cost function agrees most with the old cost function at the same weight value. This holds true for all the weight value, not only the default (0.7).  ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
