[
https://issues.apache.org/jira/browse/SPARK-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116616#comment-14116616
]
Patrick Wendell commented on SPARK-3324:
----------------------------------------
Hi,
I don't totally understand the proposal. But in general, the issue is that
these yarn versions have different API's, so you can not have a single folder
of source code that compiles against both YARN API's. So if you mean the
profile would simply influence the version of YARN we compile against, that's
not sufficient. Maybe you have something more elegant in mind? If you could
produce a small example of how you'd do this using profiles, I'd be happy to
take a look at it and we can see if it's better or worse than what we have now
:)
Our current approach does have the issue of being IDE-unfriendly.
> YARN module has nonstandard structure which cause compile error In IntelliJ
> ---------------------------------------------------------------------------
>
> Key: SPARK-3324
> URL: https://issues.apache.org/jira/browse/SPARK-3324
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.1.0
> Environment: Mac OS: 10.9.4
> IntelliJ IDEA: 13.1.4
> Scala Plugins: 0.41.2
> Maven: 3.0.5
> Reporter: Yi Tian
> Priority: Minor
> Labels: intellij, maven, yarn
>
> The YARN module has nonstandard path structure like:
> {code}
> ${SPARK_HOME}
> |--yarn
> |--alpha (contains yarn api support for 0.23 and 2.0.x)
> |--stable (contains yarn api support for 2.2 and later)
> | |--pom.xml (spark-yarn)
> |--common (Common codes not depending on specific version of Hadoop)
> |--pom.xml (yarn-parent)
> {code}
> When we use maven to compile yarn module, maven will import 'alpha' or
> 'stable' module according to profile setting.
> And the submodule like 'stable' use the build propertie defined in
> yarn/pom.xml to import common codes to sourcePath.
> It will cause IntelliJ can't directly recognize sources in common directory
> as sourcePath.
> I thought we should change the yarn module to a unified maven jar project,
> and add specify different version of yarn api via maven profile setting.
> It will resolve the compile error in IntelliJ and make the yarn module more
> simple and clear.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
