[ https://issues.apache.org/jira/browse/HIVE-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116017#comment-15116017 ]
Hive QA commented on HIVE-12629: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12784193/HIVE-12629.1-spark.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9867 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_lateral_view_mapjoin org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/1039/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/1039/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-1039/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12784193 - PreCommit-HIVE-SPARK-Build > hive.auto.convert.join=true makes lateral view join sql failed on spark > engine on yarn > -------------------------------------------------------------------------------------- > > Key: HIVE-12629 > URL: https://issues.apache.org/jira/browse/HIVE-12629 > Project: Hive > Issue Type: Bug > Components: Spark > Affects Versions: 1.2.1 > Reporter: 吴子美 > Assignee: Chao Sun > Attachments: HIVE-12629.1-spark.patch > > > I am using hive1.2 on spark on yarn. > I found > select count(1) from > (select user_id from xxx group by user_id ) a join > (select user_id from yyy lateral view json_tuple(u, 'h') v1 as h) b > on a.user_id=b.user_id ; > failed in hive on spark on yarn, but OK in hive on MR. > I tried the following sql on spark. It was OK. > select count(1) from > (select user_id from xxx group by user_id ) a left join > (select user_id from yyy lateral view json_tuple(u, 'h') v1 as h) b > on a.user_id=b.user_id ; > When I turn hive.auto.convert.join from true to false. Everything goes OK. > The error message in hive.log was : > {code} > 2015-12-09 21:10:17,190 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - 15/12/09 21:10:17 INFO log.PerfLogger: > <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities> > 2015-12-09 21:10:17,190 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - 15/12/09 21:10:17 INFO exec.Utilities: > Serializing ReduceWork via kryo > 2015-12-09 21:10:17,214 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - 15/12/09 21:10:17 INFO log.PerfLogger: > </PERFLOG method=serializePlan start=1449666617190 end=1449666617214 > duration=24 from=org.apache.hadoop.hive.ql.exec.Utilities> > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - 15/12/09 21:10:17 INFO client.RemoteDriver: > Failed to run job 8fed1ca8-834f-497f-b189-eab343440a9f > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - java.lang.IllegalStateException: Connection > already exists > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > org.apache.hadoop.hive.ql.exec.spark.SparkPlan.connect(SparkPlan.java:142) > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generateParentTran(SparkPlanGenerator.java:142) > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:106) > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:252) > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366) > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335) > 2015-12-09 21:10:17,261 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > java.util.concurrent.FutureTask.run(FutureTask.java:262) > 2015-12-09 21:10:17,262 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > 2015-12-09 21:10:17,262 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > 2015-12-09 21:10:17,262 INFO [stderr-redir-1]: client.SparkClientImpl > (SparkClientImpl.java:run(569)) - at > java.lang.Thread.run(Thread.java:745) > 2015-12-09 21:10:17,266 INFO [RPC-Handler-3]: client.SparkClientImpl > (SparkClientImpl.java:handle(522)) - Received result for > 8fed1ca8-834f-497f-b189-eab343440a9f > 2015-12-09 21:10:18,054 ERROR [HiveServer2-Background-Pool: Thread-43]: > status.SparkJobMonitor (SessionState.java:printError(960)) - Status: Failed > 2015-12-09 21:10:18,055 INFO [HiveServer2-Background-Pool: Thread-43]: > log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG > method=SparkRunJob start=1449666615051 end=1449666618055 duration=3004 > from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor> > 2015-12-09 21:10:18,076 ERROR [HiveServer2-Background-Pool: Thread-43]: > ql.Driver (SessionState.java:printError(960)) - FAILED: Execution Error, > return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask > {code} > Is it the bug of hive on spark? -- This message was sent by Atlassian JIRA (v6.3.4#6332)