Ryan Williams created SPARK-5115:
------------------------------------
Summary: Intellij fails to find hadoop classes in Spark "yarn"
modules
Key: SPARK-5115
URL: https://issues.apache.org/jira/browse/SPARK-5115
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 1.2.0
Reporter: Ryan Williams
Intellij's parsing of Spark's POMs works like a charm for the most part,
however it fails to resolve the hadoop and yarn dependencies in the Spark
{{yarn}} and {{network/yarn}} modules.
Imports and later references to imported classes show up as errors, e.g.
!http://f.cl.ly/items/0g3w3s0t45402z30011l/Screen%20Shot%202015-01-06%20at%206.42.52%20PM.png!
Opening the module settings, we see that IntelliJ is looking for version
{{1.0.4}} of [each yarn JAR that the Spark YARN module depends
on|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/yarn/pom.xml#L41-L56],
and failing to find them:
!http://f.cl.ly/items/2d320l2h2o2N1m0t2X3b/yarn.png!
This, in turn, is due to the parent POM [defaulting {{hadoop.version}} to
{{1.0.4}}|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/pom.xml#L122].
AFAIK, having the default-hadoop-version be {{1.0.4}} is not that important and
may just be an accident of history; people typically select a Maven profile
when building Spark that matches the version of Hadoop that they intend to run
with.
This suggests one possible fix: bump the default Hadoop version to >= 2. I've
tried this locally and it resolves Intellij's difficulties with the "yarn" and
"network/yarn" modules; [PR #3917|https://github.com/apache/spark/pull/3917]
does this.
Another fix would be to declare a {{hadoop.version}} property in
{{yarn/pom.xml}} and add that to the YARN dependencies in that file; [PR
#3918|https://github.com/apache/spark/pull/3918] does that.
It is more obvious to me in the former case that the existing rules that govern
what {{hadoop.version}} the YARN dependencies inherit will still apply.
For the latter, or potentially other ways to configure IntelliJ / Spark's POMs,
someone with more Maven/IntelliJ fu may need to chime in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
