[
https://issues.apache.org/jira/browse/SPARK-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Williams updated SPARK-5115:
---------------------------------
Description:
Intellij's parsing of Spark's POMs works like a charm for the most part,
however it fails to resolve the hadoop and yarn dependencies in the Spark
{{yarn}} and {{network/yarn}} modules.
Imports and later references to imported classes show up as errors, e.g.
!http://f.cl.ly/items/0g3w3s0t45402z30011l/Screen%20Shot%202015-01-06%20at%206.42.52%20PM.png!
Opening the module settings, we see that IntelliJ is looking for version
{{1.0.4}} of [each yarn JAR that the Spark YARN module depends
on|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/yarn/pom.xml#L41-L56],
and failing to find them:
!http://f.cl.ly/items/2d320l2h2o2N1m0t2X3b/yarn.png!
This, in turn, is due to the parent POM [defaulting {{hadoop.version}} to
{{1.0.4}}|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/pom.xml#L122].
AFAIK, having the default-hadoop-version be {{1.0.4}} is not that important and
may just be an accident of history; people typically select a Maven profile
when building Spark that matches the version of Hadoop that they intend to run
with.
This suggests one possible fix: bump the default Hadoop version to >= 2. I've
tried this locally and it resolves Intellij's difficulties with the "yarn" and
"network/yarn" modules; [PR #3917|https://github.com/apache/spark/pull/3917]
does this.
Another fix would be to declare a {{hadoop.version}} property in
{{yarn/pom.xml}} and add that to the YARN dependencies in that file; [PR
#3918|https://github.com/apache/spark/pull/3918] does that.
It is more obvious to me in the former case that the existing rules that govern
what {{hadoop.version}} the YARN dependencies inherit will still apply.
For the latter, or potentially other ways to configure IntelliJ / Spark's POMs
to address this issue, someone with more Maven/IntelliJ fu may need to chime in.
was:
Intellij's parsing of Spark's POMs works like a charm for the most part,
however it fails to resolve the hadoop and yarn dependencies in the Spark
{{yarn}} and {{network/yarn}} modules.
Imports and later references to imported classes show up as errors, e.g.
!http://f.cl.ly/items/0g3w3s0t45402z30011l/Screen%20Shot%202015-01-06%20at%206.42.52%20PM.png!
Opening the module settings, we see that IntelliJ is looking for version
{{1.0.4}} of [each yarn JAR that the Spark YARN module depends
on|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/yarn/pom.xml#L41-L56],
and failing to find them:
!http://f.cl.ly/items/2d320l2h2o2N1m0t2X3b/yarn.png!
This, in turn, is due to the parent POM [defaulting {{hadoop.version}} to
{{1.0.4}}|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/pom.xml#L122].
AFAIK, having the default-hadoop-version be {{1.0.4}} is not that important and
may just be an accident of history; people typically select a Maven profile
when building Spark that matches the version of Hadoop that they intend to run
with.
This suggests one possible fix: bump the default Hadoop version to >= 2. I've
tried this locally and it resolves Intellij's difficulties with the "yarn" and
"network/yarn" modules; [PR #3917|https://github.com/apache/spark/pull/3917]
does this.
Another fix would be to declare a {{hadoop.version}} property in
{{yarn/pom.xml}} and add that to the YARN dependencies in that file; [PR
#3918|https://github.com/apache/spark/pull/3918] does that.
It is more obvious to me in the former case that the existing rules that govern
what {{hadoop.version}} the YARN dependencies inherit will still apply.
For the latter, or potentially other ways to configure IntelliJ / Spark's POMs,
someone with more Maven/IntelliJ fu may need to chime in.
> Intellij fails to find hadoop classes in Spark "yarn" modules
> -------------------------------------------------------------
>
> Key: SPARK-5115
> URL: https://issues.apache.org/jira/browse/SPARK-5115
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Ryan Williams
>
> Intellij's parsing of Spark's POMs works like a charm for the most part,
> however it fails to resolve the hadoop and yarn dependencies in the Spark
> {{yarn}} and {{network/yarn}} modules.
> Imports and later references to imported classes show up as errors, e.g.
> !http://f.cl.ly/items/0g3w3s0t45402z30011l/Screen%20Shot%202015-01-06%20at%206.42.52%20PM.png!
> Opening the module settings, we see that IntelliJ is looking for version
> {{1.0.4}} of [each yarn JAR that the Spark YARN module depends
> on|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/yarn/pom.xml#L41-L56],
> and failing to find them:
> !http://f.cl.ly/items/2d320l2h2o2N1m0t2X3b/yarn.png!
> This, in turn, is due to the parent POM [defaulting {{hadoop.version}} to
> {{1.0.4}}|https://github.com/apache/spark/blob/bb38ebb1abd26b57525d7d29703fd449e40cd6de/pom.xml#L122].
> AFAIK, having the default-hadoop-version be {{1.0.4}} is not that important
> and may just be an accident of history; people typically select a Maven
> profile when building Spark that matches the version of Hadoop that they
> intend to run with.
> This suggests one possible fix: bump the default Hadoop version to >= 2. I've
> tried this locally and it resolves Intellij's difficulties with the "yarn"
> and "network/yarn" modules; [PR
> #3917|https://github.com/apache/spark/pull/3917] does this.
> Another fix would be to declare a {{hadoop.version}} property in
> {{yarn/pom.xml}} and add that to the YARN dependencies in that file; [PR
> #3918|https://github.com/apache/spark/pull/3918] does that.
> It is more obvious to me in the former case that the existing rules that
> govern what {{hadoop.version}} the YARN dependencies inherit will still apply.
> For the latter, or potentially other ways to configure IntelliJ / Spark's
> POMs to address this issue, someone with more Maven/IntelliJ fu may need to
> chime in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
