Hi all, I¹m having hard time trying to find out ways to report exception that happens during computation to the end-user of Spark system without having them ssh into the worker nodes or accessing Spark UI. For example, if some exception happens in the code that runs on worker nodes (e.g. IllegalStateException due to wrong user input), SparkContext only shows the following vague exception, and I¹d have to dig into the worker node to get the actual exception.
> Exception saving /tmp/data/dTableIL.dtconfig: org.apache.spark.SparkException: > Job failed: Task 343.0:8 failed more than 4 times > org.apache.spark.SparkException: Job failed: Task 343.0:8 failed more than 4 > times > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSchedul > er.scala:760) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSchedul > er.scala:758) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:758) > at > org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:379) > at > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSchedule > r$$run(DAGScheduler.scala:441) > at org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:149) Is there any way to forward the exception to SparkContext? If not, what are some work-arounds that can mitigate the problem here? Thanks in advance! Mingyu
smime.p7s
Description: S/MIME cryptographic signature
