Hi Hoc Phan
The stack is full of permission denied, so I would look in that direction
first.
Could you try running only the:
myfile = sc.textFile("hdfs://cdhe1master.fbdl.local:8020/
user/zeppelin/testcdh.log")
On Thu, Dec 17, 2015 at 6:06 PM, Amos B. Elberg <amos.elb...@me.com> wrote:
> Zeppelin doesn’t set the memory. Anyhow, you have my two cents, good
> luck.
>
> From: Hoc Phan <quang...@yahoo.com> <quang...@yahoo.com>
> Reply: Hoc Phan <quang...@yahoo.com> <quang...@yahoo.com>
> Date: December 17, 2015 at 4:03:39 AM
> To: Amos B. Elberg <amos.elb...@me.com> <amos.elb...@me.com>,
> users@zeppelin.incubator.apache.org <users@zeppelin.incubator.apache.org>
> <users@zeppelin.incubator.apache.org>
>
> Subject: Re: Spark Job aborted due to stage failure - error=13
> Permission denied
>
> Hi
>
> I just followed a very simple exercise below
>
> myfile =
> sc.textFile("hdfs://cdhe1master.fbdl.local:8020/user/zeppelin/testcdh.log")
> lineLengths = myfile.map(lambda s: len(s))
> totalLength = lineLengths.reduce(lambda a, b: a + b)
>
> The file is tiny like few hundreds KB (not even 1 MB).
>
> This is out-of-the-box Cloudera 5.5 and Spark 1.5 on YARN. I am running on
> machines with 100GB RAM. So there must be something wrong either in how
> Zeppelin sets memory. Because like I said in previous thread, I do pyspark
> CLI and run the exact lines just fine (in the same machine and same user).
>
>
>
>
>
> On Thursday, December 17, 2015 12:55 AM, Amos B. Elberg <
> amos.elb...@me.com> wrote:
>
>
> If its failing at the reduce step and you’re getting an OOM error, then
> its probably not the *executor* memory that’s the issue. There’s also
> master memory and backend memory.
>
> When you do a reduce(), Spark wants to dump the data into the Spark
> “backend” which is the JVM process that initiated the spark job. That’s
> probably sitting on the same machine as your zeppelin server. Anyway, if
> its a lot of data, your algorithm can run fine on the executors but OOM on
> the reduce when all the data hits the backend.
>
> What you want to do is look at your logs closely and try to figure out if
> the stack traces you’re seeing are actually coming from executory processes
> or the backend or master.
>
> You can increase the memory for the backend by configuring
> spark.driver.memory.
>
> Other alternatives are to change your algorithm, such as with
> reduceByKey(), so the reduce step happens in chunks on the executors rather
> than on the backend.
>
> But, just a warning — whenever I’ve had issues of OOM’ing the backend,
> trying to fix it by adjusting memory settings always turned out to be a
> rabbit-hole. So, you could also interpret the error as a yellow-flag that
> you should re-engineer your algorithm. That’s what I do when I see the
> error now.
>
>
> From: Hoc Phan <quang...@yahoo.com> <quang...@yahoo.com>
> Reply: users@zeppelin.incubator.apache.org
> <users@zeppelin.incubator.apache.org>
> <users@zeppelin.incubator.apache.org>, Hoc Phan <quang...@yahoo.com>
> <quang...@yahoo.com>
> Date: December 17, 2015 at 3:33:15 AM
> To: users@zeppelin.incubator.apache.org
> <users@zeppelin.incubator.apache.org>
> <users@zeppelin.incubator.apache.org>, Hoc Phan <quang...@yahoo.com>
> <quang...@yahoo.com>
> Subject: Re: Spark Job aborted due to stage failure - error=13
> Permission denied
>
> Hi
>
> Any help on this? I am stuck for a week.
>
> I have tried to follow this thread
>
> http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-users/201506.mbox/%3CCABU7W=ZwKPyuPYzTQncg9wCSAs-v=c1c+welsvzx4qj7eg-...@mail.gmail.com%3E
>
> I set these without luck:
>
> > export PYTHONPATH="${SPARK_HOME}/python:${SPARK_HOME
> > }/python/lib/py4j-0.8.2.1-src.zip"
> >
> > export SPARK_YARN_USER_ENV="PYTHONPATH=${PYTHONPATH}"
> >
>
> It only failed at .reduce() step. I think it is trying to do some IO at
> /usr/bin in one of the Cloudera worker nodes. Why is that?
>
> But in the log, I also saw:
>
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> So I am quite confused. I set executor memory to 5g and that didn't help.
>
>
>
>
>
> On Tuesday, December 15, 2015 10:34 AM, Hoc Phan <quang...@yahoo.com>
> wrote:
>
>
> Hi
>
> /usr/bin is where pyspark and spark-shell is located. But all have
> executable permission.
> The trouble I don't get is when I ssh into the machine and login as
> "zeppelin" user, I was able to go through the same script in pyspark. So my
> question is what Zeppelin is trying to access? Using what user? What is a
> way to trace and troubleshoot?
>
>
>
> On Tuesday, December 15, 2015 9:46 AM, Felix Cheung <
> felixcheun...@hotmail.com> wrote:
>
>
> It looks like it doesn't have permission to launch something
>
>
> Caused by: java.io.IOException: Cannot run program "/usr/bin": error=13,
> Permission denied
>
>
> Perhaps the file path is incorrect? It looks to point to /usr/bin which is
> likely a directory.
>
>
>
>
> On Mon, Dec 14, 2015 at 12:25 PM -0800, "Hoc Phan" <quang...@yahoo.com>
> wrote:
>
> Hi
>
> When I installed Zeppelin, I created a zeppelin user with below permission:
> uid=500(zeppelin) gid=490(hdfs)
> groups=490(hdfs),492(hadoop),501(supergroup)
>
> I ran this via pyspark just fine under this zeppelin user
>
> myfile =
> sc.textFile("hdfs://cdhe1master.fbdl.local:8020/user/zeppelin/testcdh.log")
> lineLengths = myfile.map(lambda s: len(s))
> totalLength = lineLengths.reduce(lambda a, b: a + b)
>
> print totalLength
>
> However, when I run the same thing using Zeppelin, I got this error below.
> Any idea?
>
> Py4JJavaError: An error occurred while calling
> z:org.apache.spark.api.python.PythonRDD.collectAndServe.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage
> 1.0 (TID 14, cdhe1worker0.fbdl.local): java.io.IOException: Cannot run
> program "/usr/bin": error=13, Permission denied
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
> at
> org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:160)
> at
> org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:86)
> at
> org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:73)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: error=13, Permission denied
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
> at java.lang.ProcessImpl.start(ProcessImpl.java:130)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
> ... 13 more
>
> Driver stacktrace:
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1294)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1282)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1281)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1281)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
> at scala.Option.foreach(Option.scala:236)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1507)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1469)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921)
> at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:905)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
> at org.apache.spark.rdd.RDD.collect(RDD.scala:904)
> at
> org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:373)
> at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
> at py4j.Gateway.invoke(Gateway.java:259)
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
> at py4j.GatewayConnection.run(GatewayConnection.java:207)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.IOException: Cannot run program "/usr/bin": error=13,
> Permission denied
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
> at
> org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:160)
> at
> org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:86)
> at
> org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
> at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:135)
> at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:73)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: error=13, Permission denied
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
> at java.lang.ProcessImpl.start(ProcessImpl.java:130)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
> ... 13 more
>
> (<class 'py4j.protocol.Py4JJavaError'>, Py4JJavaError(u'An error occurred
> while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.\n',
> JavaObject id=o87), <traceback object at 0x1d37b00>)
>
>
>
>
>
>
>
