That seems the best options as of now since the URL can be used to access the actual Spark UI and that should have the most information.
Bikas ________________________________ From: Vivek Suvarna <[email protected]> Sent: Friday, August 11, 2017 3:20:46 AM To: [email protected] Subject: Re: Propagating pyspark errors to Livy I'm currently getting the tracking url and giving it back to the calling program. Guess that's the best option then. Thanks Sent from my iPhone On 11 Aug 2017, at 2:55 PM, Saisai Shao <[email protected]<mailto:[email protected]>> wrote: I think you should check Spark application log to see the details, it is hard for Livy to get actual error from Spark. On Fri, Aug 11, 2017 at 12:03 PM, Vivek <[email protected]<mailto:[email protected]>> wrote: Hi, Is there anyway to propagate errors from pyspark back to the calling program via Livy? Currently the Livy logs only tells me that the batch job has failed. How do I get the actual error on the spark side. Regards Vivek Sent from my iPhone
