dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-642866772
https://github.com/apache/spark/pull/28788 is a PR that truns
`spark.yarn.populateHadoopClasspath` to `false` by default for `with-hadoop`
build.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621451320
Thanks all for reviews. I added a new UT, and after test passes, I'll merge
it into master, branch-3.0, branch-2.4.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-621450856
@tgravescs I plan to bring this to 2.4.6 since it's default to true and
there is no risk.
Many Spark users who use vendor's hadoop such as CDH or HDP are stuck on
very
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620738022
@tgravescs the standard "with hadoop" Spark built with Yarn is sufficient to
work in our CDH env. Thanks for the review. I'll update the doc and config.md.
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620404585
Ping @jerryshao @tgravescs @ulysses-you who work on this area for more eyes.
This is an automated message from
dbtsai commented on pull request #28376:
URL: https://github.com/apache/spark/pull/28376#issuecomment-620296694
The other way we can do is changing `yarn.application.classpath` and
`mapreduce.application.classpath` to an empty folder, but this is more hackey
unfortunately.
