Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3943#discussion_r23246042
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
@@ -223,4 +225,43 @@ object YarnSparkHadoopUtil {
)
}
+ /**
+ * Expand environment variable using Yarn API.
+ * If environment.$$() is implemented, return the result of it.
+ * Otherwise, return the result of environment.$()
+ * Note: $$() is added in Hadoop 2.4.
+ */
+ def expandEnvironment(environment: Environment): String = {
+ var result = environment.$()
+
+ // We use reflection in order not to fail building with Hadoop 2.3 or
before.
+ val clazz = classOf[Environment]
--- End diff --
This is probably not a big deal, since this is not in any hot path, but it
seems wasteful to look for the right method every time this method is called
instead of caching the method to be called. I'd do this:
private lazy val expandMethod =
Try(classOf[Environment].getMethod("$$"))
.getOrElse(classOf[Environment].getMethod("$"))
Then your method becomes just:
def expandEnvironment(environment: Environment): String = \
expandMethod.invoke(environment).asInstanceOf[String]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
