For the first 3 errors, I figured out that JAVA_HOME was not set on my machine. Logs just say exit code 127 returned by the driver process. I had to instrument some code to print the output/error stream of the driver process launched by ContextLauncher. Only then I realized that dev/spark/bin/spark-submit is expecting JAVA_HOME to be set.
The second error, TestSparkClient.testJobSubmission, is random. I see the failure ~25% of the times. As it turned out, the Echo job was returning, sometimes, even before we added the listener on the job handle. So the listener.onJobStarted method is not invoked. This can be fixed in multiple ways but I updated the Echo job, which used to simply return its attribute value, to do what ScalaEcho job is doing. So the Echo job now returns "jc.sc().parallelize(list, 1).collect().get(0);". Even this could still be fast enough to cause the same error. However I couldn't reproduce it again. Also the updated code is taking ~1 sec or more on spark's side. So hopefully this should be ok. Thoughts? -Rahul On Sat, Apr 4, 2020 at 2:24 AM rahul challapalli <challapallira...@gmail.com> wrote: > Hi Team, > > When I tried to build master using "mvn clean test -Drat.skip=true > -DskipITs", I got unit test failures in rsc module. SPARK_HOME is being set > to "dev/spark" which only contains spark-submit but none of the spark > related jar file. So I am not sure how spark-submit is supposed to work. Is > it expected to load spark jars from the classpath since spark is a > dependency to compile livy? In any case the below 3 tests failed. > > Tests in error: > > TestSparkClient.testConnectToRunningContext:338->runTest:575->runTest:588 » > Execution > TestSparkClient.testImpersonation:321->runTest:575->runTest:588 » > Execution ja... > TestSparkClient.testRemoteClient:213->runTest:575->runTest:588 » > Execution jav... > > I tried to point SPARK_HOME to a downloaded spark tar ball (spark-2.3) and > the below tests passed.....but another test failed in the same rsc module > > Failed tests: > TestSparkClient.testJobSubmission:88->runTest:575->runTest:590 > Wanted but not invoked: > listener.onJobStarted( > org.apache.livy.rsc.JobHandleImpl@6e22d6bf > ); > -> at org.apache.livy.rsc.TestSparkClient$1.call(TestSparkClient.java:101) > > However, there were other interactions with this mock: > -> at > org.apache.livy.client.common.AbstractJobHandle.fireStateChange(AbstractJobHandle.java:97) > > > Any idea what I could be missing? > > -Rahul > > > >
