Yeah, only using numpy. Strange, it must be an issue with my setup. Will let you know if I figure it out.
-Sandy On Fri, Dec 20, 2013 at 6:03 AM, Michael Ronquest <[email protected]> wrote: > Sandy, > Are you just using numpy? numexpr (fast math for numpy arrays) has > issues on workers. > Cheers, > Mike > > On 12/19/2013 06:04 PM, Sandy Ryza wrote: > >> Verified that python is installed on the worker. When I simplify my job >> I'm able to to get more stuff in stderr, but it's just the Java log4j >> messages. >> >> I narrowed it down and I'm pretty sure the error is coming from my use of >> numpy - I'm trying to pass around records that hold numpy arrays. I've >> verified that numpy is installed on the workers and that the job works >> locally on the master. Is there anything else I need to do for accessing >> numpy from workers? >> >> thanks, >> Sandy >> >> >> >> On Thu, Dec 19, 2013 at 2:23 PM, Matei Zaharia >> <[email protected]<mailto: >> [email protected]>> wrote: >> >> It might also mean you don’t have Python installed on the worker. >> >> On Dec 19, 2013, at 1:17 PM, Jey Kottalam <[email protected] >> <mailto:[email protected]>> wrote: >> >> > That's pretty unusual; normally the executor's stderr output would >> > contain a stacktrace and any other error messages from your Python >> > code. Is it possible that the PySpark worker crashed in C code >> or was >> > OOM killed? >> > >> > On Thu, Dec 19, 2013 at 11:10 AM, Sandy Ryza >> <[email protected] <mailto:[email protected]>> wrote: >> >> Hey All, >> >> >> >> Where are python logs in PySpark supposed to go? My job is >> getting a >> >> org.apache.spark.SparkException: Python worker exited >> unexpectedly (crashed) >> >> but when I look at the stdout/stderr logs in the web UI, >> nothing interesting >> >> shows up (stdout is empty and stderr just has the spark >> executor command). >> >> >> >> Is this the expected behavior? >> >> >> >> thanks in advance for any guidance, >> >> Sandy >> >> >> >
