Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6083#discussion_r30160116
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -371,6 +371,13 @@ class SparkContext(config: SparkConf) extends Logging
with ExecutorAllocationCli
throw new SparkException("An application name must be set in your
configuration")
}
+ // Thread name has been set to "Driver" if user code ran by AM on a
YARN cluster
--- End diff --
Two things here:
- Should also check for `yarn-standalone` since the code still accepts that
as an alias for `yarn-cluster`.
- I think the check for `Driver` is too brittle. I think checking for
`spark.yarn.app.id` in `SparkConf` is slightly better. Both are not 100%
fool-proof since users can muck with them, but at least the latter is an
internal API (i.e. the YARN code depends on that property to exist for other
things).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
