heyang wang created ZEPPELIN-2589:
-------------------------------------
Summary: Let Livy interpreter show traceback log
Key: ZEPPELIN-2589
URL: https://issues.apache.org/jira/browse/ZEPPELIN-2589
Project: Zeppelin
Issue Type: Improvement
Components: livy-interpreter
Affects Versions: 0.7.1
Environment: zeppelin-0.7.1,livy-0.4-snapshot
Reporter: heyang wang
I am using Livy interpreter to interact with spark cluster. However, every time
some error occurs,Zeppelin only print one line of error message which is
extremely unhelpful to debug the code.
For example, If I get the following error output in Zeppelin
<console>:113: error: overloaded method value createDataFrame with alternatives:
I can't found out exactly what caused the error.
But executing the same code on spark shell I would get the following error
<console>:56: error: overloaded method value createDataFrame with alternatives:
(data: java.util.List[_],beanClass: Class[_])org.apache.spark.sql.DataFrame
<and>
(rdd: org.apache.spark.api.java.JavaRDD[_],beanClass:
Class[_])org.apache.spark.sql.DataFrame <and>
(rdd: org.apache.spark.rdd.RDD[_],beanClass:
Class[_])org.apache.spark.sql.DataFrame <and>
(rows: java.util.List[org.apache.spark.sql.Row],schema:
org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame <and>
(rowRDD: org.apache.spark.api.java.JavaRDD[org.apache.spark.sql.Row],schema:
org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame <and>
(rowRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row],schema:
org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame
cannot be applied to (org.apache.spark.rdd.RDD[String],
org.apache.spark.sql.types.StructType)
val testDF = spark.createDataFrame(rdds, schema)
All the detailed error log is returned by Livy server, but in another field
called "traceback". In BaseLivyInterpreter.java, Zepplein only print out the
"evalue" which contain only the header line of the error log.
I think print "traceback" log would be much more helpful to debug spark job
when using Livy.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
