[
https://issues.apache.org/jira/browse/OOZIE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15456664#comment-15456664
]
Peter Cseh commented on OOZIE-2554:
-----------------------------------
I've looked into this today and got stuck.
First, Spark wanted to have {{YARN_CONF_DIR}} env. varialbe to be set. As the
MiniMRCluster runs in the same JVM as the test, this is not possible. I managed
to work around this by setting the {{spark.testing}} property in TestSparkMain:
{code}
System.setProperty("spark.testing", "true");
{code}
But now, I get the error:
{noformat}
Call From GezapetiMBP.local/192.168.1.4 to 0.0.0.0:8032 failed on connection
exception: java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
{noformat}
This is because Spark still tries to get the Yarn configuration from that
folder, I guess this is the fallback value.
I've checked the Spark tests and they are using
[SparkLauncher|https://github.com/apache/spark/blob/branch-1.6/launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java],
where it's possible to include environment variables for the Spark process it
spawns.
{{SparkMain}} might be changed to use {{SparkLauncher}}, but I rather replace
the {{SparkSubmit.main}} call to a dynamic way to load and call Spark and we
could replace it to have a thing to validate the parameters given to Spark in
the tests.
Any thoughts on this, [~satishsaley]?
> Spark action tests on yarn-cluster, yarn-client mode
> ----------------------------------------------------
>
> Key: OOZIE-2554
> URL: https://issues.apache.org/jira/browse/OOZIE-2554
> Project: Oozie
> Issue Type: New Feature
> Reporter: Satish Subhashrao Saley
> Assignee: Satish Subhashrao Saley
>
> Currently, we have unit test cases for Spark Action in local mode. To verify
> other functionality, we should have tests in yarn-cluster and yarn-client
> mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
