[
https://issues.apache.org/jira/browse/SPARK-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267035#comment-14267035
]
Apache Spark commented on SPARK-5115:
-------------------------------------
User 'ryan-williams' has created a pull request for this issue:
https://github.com/apache/spark/pull/3917
> Intellij fails to find hadoop classes in Spark "yarn" modules
> -------------------------------------------------------------
>
> Key: SPARK-5115
> URL: https://issues.apache.org/jira/browse/SPARK-5115
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Ryan Williams
>
> Intellij's parsing of Spark's POMs works like a charm for the most part,
> however it fails to resolve the hadoop and yarn dependencies in the Spark
> {{yarn}} and {{network/yarn}} modules.
> Imports and later references to imported classes show up as errors, e.g.
> !http://f.cl.ly/items/0g3w3s0t45402z30011l/Screen%20Shot%202015-01-06%20at%206.42.52%20PM.png!
> Opening the module settings, we see that IntelliJ is looking for version
> {{1.0.4}} of [each yarn JAR that the Spark YARN module depends
> on|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/yarn/pom.xml#L41-L56],
> and failing to find them:
> !http://f.cl.ly/items/2d320l2h2o2N1m0t2X3b/yarn.png!
> This, in turn, is due to the parent POM [defaulting {{hadoop.version}} to
> {{1.0.4}}|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/pom.xml#L122].
> AFAIK, having the default-hadoop-version be {{1.0.4}} is not that important
> and may just be an accident of history; people typically select a Maven
> profile when building Spark that matches the version of Hadoop that they
> intend to run with.
> This suggests one possible fix: bump the default Hadoop version to >= 2. I've
> tried this locally and it resolves Intellij's difficulties with the "yarn"
> and "network/yarn" modules; [PR
> #3917|https://github.com/apache/spark/pull/3917] does this.
> Another fix would be to declare a {{hadoop.version}} property in
> {{yarn/pom.xml}} and add that to the YARN dependencies in that file; [PR
> #3918|https://github.com/apache/spark/pull/3918] does that.
> It is more obvious to me in the former case that the existing rules that
> govern what {{hadoop.version}} the YARN dependencies inherit will still apply.
> For the latter, or potentially other ways to configure IntelliJ / Spark's
> POMs, someone with more Maven/IntelliJ fu may need to chime in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
