Github user sathiyapk commented on the issue:
https://github.com/apache/spark/pull/19295
@gatorsmile thanks for your comments. Here are my thoughts, thanks for
correcting me if i'm wrong. (sorry for the big comment though :))
1. This PR don't change any existing API, it adds a new one.
2. In the usual cases, for the people who don't use `ExperimentalMethods`,
it don't affect or break anything.
3. For the people who use `ExperimentalMethods`, irrespective of whether it
is pre-optimizer or post-optimizer rule, it will break anyway if they do it
wrong.
4. One of the advantages of this PR
`sparkSession.experimental.extraPreOptimizations` is that the user provided
rule can get further optimizer by the native rules of spark, which is not
possible with `sparkSession.experimental.extraOptimizations`. I'm writing a
blog post regarding this with an example, i will post the link soon.
5. Last but not least, one of the main intention of the spark catalyst
optimizer, as mentioned in its sigmod paper, is it's simplicity in defining new
optimization rules and plug it into the query optimizer during runtime, so we
should consider not to limit it even if it only concerns a rare case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
