Verified that python is installed on the worker. When I simplify my job I'm able to to get more stuff in stderr, but it's just the Java log4j messages.
I narrowed it down and I'm pretty sure the error is coming from my use of numpy - I'm trying to pass around records that hold numpy arrays. I've verified that numpy is installed on the workers and that the job works locally on the master. Is there anything else I need to do for accessing numpy from workers? thanks, Sandy On Thu, Dec 19, 2013 at 2:23 PM, Matei Zaharia <[email protected]>wrote: > It might also mean you don’t have Python installed on the worker. > > On Dec 19, 2013, at 1:17 PM, Jey Kottalam <[email protected]> wrote: > > > That's pretty unusual; normally the executor's stderr output would > > contain a stacktrace and any other error messages from your Python > > code. Is it possible that the PySpark worker crashed in C code or was > > OOM killed? > > > > On Thu, Dec 19, 2013 at 11:10 AM, Sandy Ryza <[email protected]> > wrote: > >> Hey All, > >> > >> Where are python logs in PySpark supposed to go? My job is getting a > >> org.apache.spark.SparkException: Python worker exited unexpectedly > (crashed) > >> but when I look at the stdout/stderr logs in the web UI, nothing > interesting > >> shows up (stdout is empty and stderr just has the spark executor > command). > >> > >> Is this the expected behavior? > >> > >> thanks in advance for any guidance, > >> Sandy > >
