Marco Gaido created LIVY-571:
--------------------------------
Summary: When an exception happens running a query, we should
report it to end user
Key: LIVY-571
URL: https://issues.apache.org/jira/browse/LIVY-571
Project: Livy
Issue Type: Bug
Reporter: Marco Gaido
When a query fails with an exception on livy thriftserver, instead of reporting
this exception to the end user, a meaningless one is reported. Eg, with Hive
support not enabled on spark, the following query causes:
{code}
0: jdbc:hive2://localhost:10090/> create table test as select a.* from (select
1, "2") a;
Error: java.lang.RuntimeException: java.util.NoSuchElementException: Statement
820bb5c2-018b-46ea-9b7f-b0e3b9c31c46 not found in session
acf3712b-1f08-4111-950f-559fc3f3f10c.
org.apache.livy.thriftserver.session.ThriftSessionState.statementNotFound(ThriftSessionState.java:118)
org.apache.livy.thriftserver.session.ThriftSessionState.cleanupStatement(ThriftSessionState.java:107)
org.apache.livy.thriftserver.session.CleanupStatementJob.call(CleanupStatementJob.java:43)
org.apache.livy.thriftserver.session.CleanupStatementJob.call(CleanupStatementJob.java:26)
org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
{code}
Looking at the logs, of course the real problem is:
{code}
19/03/23 10:40:32 ERROR LivyExecuteStatementOperation: Error running hive
query:
org.apache.hive.service.cli.HiveSQLException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.spark.sql.AnalysisException: Hive support is required to CREATE Hive
TABLE (AS SELECT);;
'CreateTable `test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
ErrorIfExists
+- Project [1#1, 2#2]
+- SubqueryAlias `a`
+- Project [1 AS 1#1, 2 AS 2#2]
+- OneRowRelation
org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:392)
org.apache.spark.sql.execution.datasources.HiveOnlyCheck$$anonfun$apply$12.apply(rules.scala:390)
org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:117)
org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:390)
org.apache.spark.sql.execution.datasources.HiveOnlyCheck$.apply(rules.scala:388)
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$2.apply(CheckAnalysis.scala:386)
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:386)
org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
org.apache.livy.thriftserver.session.SqlJob.executeSql(SqlJob.java:74)
org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:64)
org.apache.livy.thriftserver.session.SqlJob.call(SqlJob.java:35)
org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
at
org.apache.livy.thriftserver.LivyExecuteStatementOperation.execute(LivyExecuteStatementOperation.scala:147)
at
org.apache.livy.thriftserver.LivyExecuteStatementOperation$$anon$1$$anon$2.run(LivyExecuteStatementOperation.scala:97)
at
org.apache.livy.thriftserver.LivyExecuteStatementOperation$$anon$1$$anon$2.run(LivyExecuteStatementOperation.scala:94)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at
org.apache.livy.thriftserver.LivyExecuteStatementOperation$$anon$1.run(LivyExecuteStatementOperation.scala:107)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
....
{code}
And this should be reported to the end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
