Would you please provide more information about how you create a Livy session? As for now, Livy only supports spark on yarn and local mode officially, we don't test on standalone cluster mode, so maybe there's some issues in it.
On Mon, Oct 16, 2017 at 4:29 AM, Junaid Nasir <[email protected]> wrote: > Hi everyone, > I am using livy (livy-0.4.0-incubating-bin), and a remote spark > (spark-2.2.0-bin-hadoop2.7) stand alone cluster. livy's setting is default > except setting livy.spark.master = spark://10.128.1.1:6066 and > livy.spark.deploy-mode = cluster (have tried with default deploy-mode but > then it throws a different error) > when i try to create an interactive spark session on remote cluster it > creates the driver and immediately sends it to error state. > > livy console error > > 17/10/15 20:18:08 INFO LineBufferedStream: stdout: 17/10/15 > 20:18:08 INFO RestSubmissionClient: Submitting a request to launch an > application in spark://10.128.1.1:6066. > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO > RestSubmissionClient: Submission successfully created as > driver-20171015201808-0002. Polling submission state... > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO > RestSubmissionClient: Submitting a request for the status of submission > driver-20171015201808-0002 in spark://10.128.1.1:6066. > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO > RestSubmissionClient: State of driver driver-20171015201808-0002 is now ERROR. > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO > RestSubmissionClient: Driver is running on worker > worker-20171015195836-10.128.1.1-38097 at 10.128.1.1:38097. > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 ERROR > RestSubmissionClient: Exception from the cluster: > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > java.nio.file.NoSuchFileException: spark-internal > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > java.nio.file.Files.copy(Files.java:1274) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:625) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > org.apache.spark.util.Utils$.copyFile(Utils.scala:596) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > org.apache.spark.util.Utils$.doFetchFile(Utils.scala:681) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > org.apache.spark.util.Utils$.fetchFile(Utils.scala:480) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: > org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92) > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO > RestSubmissionClient: Server responded with CreateSubmissionResponse: > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: { > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: "action" : > "CreateSubmissionResponse", > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: "message" : "Driver > successfully submitted as driver-20171015201808-0002", > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: "serverSparkVersion" : > "2.2.0", > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: "submissionId" : > "driver-20171015201808-0002", > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: "success" : true > 17/10/15 20:18:09 INFO LineBufferedStream: stdout: } > > [image: Mixmax] > <https://mixmax.com/r/592fe3b429b79365389d2354> Not using Mixmax yet? > <https://mixmax.com/r/592fe3b429b79365389d2354> > > can anyone please guide me how to resolve this? > > I have tried setting up spark locally and it works fine. so i guess > problem is with uploading jar files to remote cluster. I am using GCE for > both spark cluster and livy server but they have all ports (on internal > network open) > > spark worker log show this too > > 17/10/15 20:18:08 INFO Worker: Asked to launch driver > driver-20171015201808-0002 > 17/10/15 20:18:08 INFO DriverRunner: Copying user jar spark-internal to > /home/junaid.nasir/spark-2.2.0-bin-hadoop2.7/work/driver-20171015201808-0002/spark-internal > 17/10/15 20:18:08 INFO Utils: Copying /spark-internal to > /home/junaid.nasir/spark-2.2.0-bin-hadoop2.7/work/driver-20171015201808-0002/spark-internal > 17/10/15 20:18:08 INFO DriverRunner: Killing driver process! > 17/10/15 20:18:08 WARN Worker: Driver driver-20171015201808-0002 failed with > unrecoverable exception: java.nio.file.NoSuchFileException: spark-internal > > [image: Mixmax] > <https://mixmax.com/r/592fe3b429b79365389d2354> Not using Mixmax yet? > <https://mixmax.com/r/592fe3b429b79365389d2354> > >
