[
https://issues.apache.org/jira/browse/SPARK-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116667#comment-14116667
]
Sean Owen commented on SPARK-3324:
----------------------------------
Let me try to sketch what's funky about the structure. We have yarn/alpha,
yarn/common, yarn/stable. Understanding the purpose, I would expect each to be
a module, and that each has a src/ directory, and that alpha and stable depend
on common, and the Spark parent activates either yarn/alpha or yarn/stable
depending on profiles. IntelliJ is fine with that.
However what we have is that yarn/ is a module. But its source is in
yarn/common. But it's a pom-only module. And yarn/alpha and yarn/stable list it
as the parent and inherit all of their source directory info and dependencies
from yarn/, which is not itself a module of code. So each compiles two source
directories defined in different places. This plus profiles confused IntelliJ
and required manual intervention.
Maybe I overlook a reason this had to be done, but rejiggering this as three
simple modules should work.
Again I imagine the question is, is it worth it versus removing yarn/alpha at
some point in the future? Because it's trivial to fix how IntelliJ reads the
POMs once by hand in the IDE.
> YARN module has nonstandard structure which cause compile error In IntelliJ
> ---------------------------------------------------------------------------
>
> Key: SPARK-3324
> URL: https://issues.apache.org/jira/browse/SPARK-3324
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.1.0
> Environment: Mac OS: 10.9.4
> IntelliJ IDEA: 13.1.4
> Scala Plugins: 0.41.2
> Maven: 3.0.5
> Reporter: Yi Tian
> Priority: Minor
> Labels: intellij, maven, yarn
>
> The YARN module has nonstandard path structure like:
> {code}
> ${SPARK_HOME}
> |--yarn
> |--alpha (contains yarn api support for 0.23 and 2.0.x)
> |--stable (contains yarn api support for 2.2 and later)
> | |--pom.xml (spark-yarn)
> |--common (Common codes not depending on specific version of Hadoop)
> |--pom.xml (yarn-parent)
> {code}
> When we use maven to compile yarn module, maven will import 'alpha' or
> 'stable' module according to profile setting.
> And the submodule like 'stable' use the build propertie defined in
> yarn/pom.xml to import common codes to sourcePath.
> It will cause IntelliJ can't directly recognize sources in common directory
> as sourcePath.
> I thought we should change the yarn module to a unified maven jar project,
> and add specify different version of yarn api via maven profile setting.
> It will resolve the compile error in IntelliJ and make the yarn module more
> simple and clear.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
