Dan Burkert has posted comments on this change. (
http://gerrit.cloudera.org:8080/8585 )
Change subject: [spark] add 'local-cluster' mode for unit test
......................................................................
Patch Set 1:
(5 comments)
I wasn't able to get the tests to run on my system, I got a bunch of error
messages like this:
12:22:13.383 [ERROR - ExecutorRunner for app-20171121122213-0000/0]
(Logging.scala:91) Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
at
org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
at
org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241)
at
org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195)
at
org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118)
at
org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
at
org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:47)
at
org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
at
org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
at
org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
at
org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
Does that ring any bells?
http://gerrit.cloudera.org:8080/#/c/8585/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala
File
java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala:
http://gerrit.cloudera.org:8080/#/c/8585/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala@69
PS1, Line 69: testMode.toLowerCase
Can be simplified to
System.getenv("TEST_MODE") match {
http://gerrit.cloudera.org:8080/#/c/8585/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala@70
PS1, Line 70: // According to documentation on spark code, this mode:
creates a Spark standalone
Might be useful to add a reference to SPARK-595.
http://gerrit.cloudera.org:8080/#/c/8585/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala@78
PS1, Line 78: conf.setMaster("local[*]")
I think it'd be better to throw an exception here so that it fails when
TEST_MODE is set without SPARK_HOME. Is there a reason I'm missing to do a
silent fallback?
http://gerrit.cloudera.org:8080/#/c/8585/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala@80
PS1, Line 80: val executorLog: String = System.getenv("EXE_LOG_PATH")
These configuration knobs are going to be hard to use, and I don't think they
will ultimately be very useful. Can we skip adding them for now? We can
always add them back later if they are necessary.
http://gerrit.cloudera.org:8080/#/c/8585/1/java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala@90
PS1, Line 90: "local" | _
can be simplified to _
--
To view, visit http://gerrit.cloudera.org:8080/8585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I07b2daad6f6883eb7425fa7132208818bbe65788
Gerrit-Change-Number: 8585
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <[email protected]>
Gerrit-Reviewer: Dan Burkert <[email protected]>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Comment-Date: Tue, 21 Nov 2017 21:13:42 +0000
Gerrit-HasComments: Yes
