DennisJLi commented on PR #47915:
URL: https://github.com/apache/spark/pull/47915#issuecomment-2315984878
Thanks for the high quality bar Dongjoon, you're right, all of the commits
on `branch-3.5` have it set to `true` and inspecting all of the zip sources
from the tags shows that's the case too.
After further investigation, the issue I was experiencing seems to be due to
JARs published by AWS on EMR differing from that in the Spark source.
Running on EMR 7.2.0, AWS states that they vend Spark 3.5.1. So I pulled
down `/usr/lib/spark/jars/spark-streaming_2.12-3.5.1-amzn-0.jar` and this is
what the Scala decompiled down into.
```
this.CAN_CHANGE_CACHED_PLAN_OUTPUT_PARTITIONING =
this.buildConf("spark.sql.optimizer.canChangeCachedPlanOutputPartitioning").internal().doc((new
StringBuilder(326)).append("Whether to forcibly enable some optimization rules
that can change the output partitioning of a cached query when executing it for
caching. If it is set to true, queries may need an extra shuffle to read the
cached data. This configuration is enabled by default. The optimization rules
enabled by this configuration ").append("are
").append(this.ADAPTIVE_EXECUTION_ENABLED().key()).append(" and
").append(this.AUTO_BUCKETED_SCAN_ENABLED().key()).append(".").toString()).version("3.2.0").booleanConf().createWithDefault(BoxesRunTime.boxToBoolean(false));
```
I'm not sure why they changed it, but it's my bad for not checking to make
sure the AWS vended JAR was the same first; I got mislead by the value on the
`master` branch being `false`. Thankfully this does explain the behavior I was
seeing.
I will change in PR to instead to include a note in the 3.5 to 4.0.0
documentation that this flag has been redisabled since I don't see that present
yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
