[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228289#comment-14228289 ] Xuefu Zhang commented on HIVE-8836: --- To summarize the above Spark related test failures: 1. custom_input_output_format.q, tracked by HIVE-8991 2. infer_bucket_sort_convert_join.q and parquet_join.q, tracked by HIVE-8992 3. mapjoin_hook.q, tracked by HIVE-8981 Above tez and MR related failrues should be unrelated. They need to be fixed in trunk. Thank everyone for making this happen. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.13-spark.patch, HIVE-8836.14-spark.patch, HIVE-8836.14-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227411#comment-14227411
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12684005/HIVE-8836.11-spark.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 7180 tests
executed
*Failed tests:*
{noformat}
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_counter
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_table_access_keys_stats
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/458/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/458/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-458/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12684005 - PreCommit-HIVE-SPARK-Build
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.10-spark.patch,
HIVE-8836.10-spark.patch, HIVE-8836.11-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch,
HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch,
additional-enable-spark-log.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227424#comment-14227424 ] Szehon Ho commented on HIVE-8836: - HIVE-8924 committed and fixed some errors. Mapjoin_hook tracked by HIVE-8981, feel free to ignore for now. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.10-spark.patch, HIVE-8836.10-spark.patch, HIVE-8836.11-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227460#comment-14227460 ] Szehon Ho commented on HIVE-8836: - Just looking through and one more quick observation, parquet_join is failing because the table parquet_jointable1_bucketed_sorted is corrupted, with 3 files but created with 1 bucket. It seems a problem with number of reducers in the insert statement that populate the table, but not sure if I will have a chance to look soon. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.10-spark.patch, HIVE-8836.10-spark.patch, HIVE-8836.11-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227797#comment-14227797
]
Brock Noland commented on HIVE-8836:
The latest rev missed some changes related to spark.home so it ran in process.
I will update shortly.
{noformat}
2014-11-27 08:14:40,046 INFO [main]: client.SparkClientImpl
(SparkClientImpl.java:startDriver(198)) - No spark.home provided, calling
SparkSubmit directly.
2014-11-27 08:14:40,046 DEBUG [main]: client.SparkClientImpl
(SparkClientImpl.java:startDriver(248)) - Running client driver with argv:
/usr/java/jdk1.7.0_45-cloudera/jre/bin/java org.apache.spark.deploy.SparkSubmit
--properties-file
/home/hiveptest/54.177.147.244-hiveptest-0/apache-svn-spark-source/itests/qtest-spark/target/tmp/spark-submit.3800861402634267765.properties
--class org.apache.hive.spark.client.RemoteDriver
/home/hiveptest/54.177.147.244-hiveptest-0/maven/org/apache/hive/hive-exec/0.15.0-SNAPSHOT/hive-exec-0.15.0-SNAPSHOT.jar
--remote
akka.tcp://73d8f0b4-dd16-4893-92f7-18a91307f929@10.232.134.247:35405/user/SparkClient-327f65bb-a5fd-43f2-917d-a5829a58465a
2014-11-27 08:14:40,134 WARN [Driver]: client.SparkClientImpl
(SparkClientImpl.java:run(264)) - Child process exited with code 1.
{noformat}
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.10-spark.patch,
HIVE-8836.10-spark.patch, HIVE-8836.11-spark.patch, HIVE-8836.12-spark.patch,
HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch,
HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch,
HIVE-8836.9-spark.patch, additional-enable-spark-log.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227799#comment-14227799
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12684069/HIVE-8836.12-spark.patch
{color:red}ERROR:{color} -1 due to 498 failed/errored test(s), 7167 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-escape_distributeby1.q-auto_sortmerge_join_7.q-bucketmapjoin_negative3.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_add_part_multiple
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join28
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_4
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227801#comment-14227801 ] Brock Noland commented on HIVE-8836: The latest rev also missed the changes to the query plan (number of reducers). I will include that in my next patch. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.10-spark.patch, HIVE-8836.10-spark.patch, HIVE-8836.11-spark.patch, HIVE-8836.12-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227812#comment-14227812 ] Brock Noland commented on HIVE-8836: bq. Just looking through and one more quick observation, parquet_join is failing because the table parquet_jointable1_bucketed_sorted is corrupted, with 3 files but created with 1 bucket. It seems a problem with number of reducers in the insert statement that populate the table, but not sure if I will have a chance to look soon. Ahh interesting. We'll probably need to fall back to the number of buckets for bucketed tables. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.10-spark.patch, HIVE-8836.10-spark.patch, HIVE-8836.11-spark.patch, HIVE-8836.12-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227878#comment-14227878
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12684085/HIVE-8836.13-spark.patch
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 7182 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_counter
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/461/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/461/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-461/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12684085 - PreCommit-HIVE-SPARK-Build
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.11-spark.patch, HIVE-8836.12-spark.patch,
HIVE-8836.13-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch,
HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch,
additional-enable-spark-log.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227883#comment-14227883 ] Brock Noland commented on HIVE-8836: Latest run looks pretty good: # One test which needs the number of reducers updated (must have just been committed) # A few failures due to instability in RSC (fixing this will be a long term project which we can do after commit IMO) # A few ordering issues (fixing in trunk via HIVE-8989) # A few other known issues like parquet_join, custom_input_output_format, and map_hook. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.11-spark.patch, HIVE-8836.12-spark.patch, HIVE-8836.13-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227904#comment-14227904 ] Xuefu Zhang commented on HIVE-8836: --- Nice. Once we have reduced the number of failures to a trackable few, we can commit this and track remaining issues in individual JIRAs. I'm sure more issues will be discovered with this on the way, but that's a good thing. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.11-spark.patch, HIVE-8836.12-spark.patch, HIVE-8836.13-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227949#comment-14227949 ] Brock Noland commented on HIVE-8836: Agreed this latest patch includes HIVE-8989. It might not apply due to the recent commits to SVN and the slow svn to git publishing. If the results look similar to last time with some improvement, I will commit the change. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.13-spark.patch, HIVE-8836.14-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227982#comment-14227982
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12684116/HIVE-8836.14-spark.patch
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 7181 tests executed
*Failed tests:*
{noformat}
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_counter
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/462/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/462/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-462/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12684116 - PreCommit-HIVE-SPARK-Build
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.13-spark.patch, HIVE-8836.14-spark.patch,
HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch,
additional-enable-spark-log.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228059#comment-14228059
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12684127/HIVE-8836.14-spark.patch
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 7182 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/463/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/463/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-463/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12684127 - PreCommit-HIVE-SPARK-Build
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.13-spark.patch, HIVE-8836.14-spark.patch,
HIVE-8836.14-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch,
HIVE-8836.9-spark.patch, additional-enable-spark-log.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226270#comment-14226270 ] Xuefu Zhang commented on HIVE-8836: --- [~ruili], I think the number of reducers changed because of the cluster changes. Previously the plan is generated with one node with 4 cores (local[4]). Now the cluster has 2 nodes and one core each. Memory configuration is also different. I guess it's hard to tweek the cluster configuration so that the same number of reducer results. For now, I think we have to go thru the list and analyze failures one by one. It's a long list, and maybe it can be divided among people so that each only take a slice of it. Briefly checking the result, it seems the failures are caused by any of the following reasons: 1. reducer number change, which is okay. 2. result diff. It could be a matter of ordering, but could be different result also. 3. test failed to run. I noticed that we are using local-cluster[2,1,2048]. Maybe we should have a more general case where one node has more than one core. Also, we may need to adjust the memory settings. Once we have a representative of a small cluster, we probably will stay with it for some time. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226461#comment-14226461 ] Brock Noland commented on HIVE-8836: I will change it to two cores and then re-generate the outputs. This should allow us to differentiate between failed tests, changes outputs, and just reducer changes. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226730#comment-14226730
]
Brock Noland commented on HIVE-8836:
Attached patch has regenerated output for queries which had a different plan
(number of reducers). It does not update the following:
*Query Result differences*
{noformat}
auto_join_without_localtask.q.out
count.q.out
join_filters_overlap.q.out
limit_pushdown.q.out
mapreduce2.q.out
multi_insert_gby3.q.out
multi_join_union.q.out
ppd_outer_join3.q.out
ptf_decimal.q.out
ptf_general_queries.q.out
smb_mapjoin_1.q.out
smb_mapjoin_2.q.out
smb_mapjoin_4.q.out
smb_mapjoin_5.q.out
smb_mapjoin_8.q.out
stats_counter.q.out
table_access_keys_stats.q.out
uniquejoin.q.out
vector_decimal_aggregate.q.out
vectorization_13.q.out
join_reorder.q.out
outer_join_ppr.q.out
*Failed*
{noformat}
bucketmapjoin1.q.out
groupby_multi_insert_common_distinct.q.out
groupby_multi_single_reducer.q.out
infer_bucket_sort_convert_join.q.out
mapjoin_hook.q.out
smb_mapjoin9
{noformat}
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch,
HIVE-8836.7-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226820#comment-14226820
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12683900/HIVE-8836.7-spark.patch
{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 7177 tests
executed
*Failed tests:*
{noformat}
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_complex_types_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_reorder
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_outer_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_decimal
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_general_queries
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_counter
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_lazy
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_uniquejoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_between_in
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_timestamp_funcs
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/449/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/449/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-449/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 48 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12683900 - PreCommit-HIVE-SPARK-Build
Enable
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226883#comment-14226883
]
Brock Noland commented on HIVE-8836:
I am making the following tests deterministic over in HIVE-8976.
{noformat}
auto_join_without_localtask.q
count.q
limit_pushdown.q
mapreduce2.q
multi_insert_gby3.q
multi_join_union.q
ppd_outer_join3.q
ptf_decimal.q
ptf_general_queries.q
{noformat}
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch,
HIVE-8836.7-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227073#comment-14227073
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12683943/HIVE-8836.9-spark.patch
{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 7178 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_gby3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_decimal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_general_queries
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_reorder
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_counter
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_uniquejoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_13
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/451/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/451/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-451/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12683943 - PreCommit-HIVE-SPARK-Build
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch,
HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227075#comment-14227075 ] Brock Noland commented on HIVE-8836: The non-spark failures are my fault due to not including the changes in HIVE-8976 for MR in this patch. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227118#comment-14227118 ] Rui Li commented on HIVE-8836: -- Thanks [~brocknoland] and [~xuefuz] for the work. Maybe we can split this job a little bit? I can take a look at the tests that failed to run. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227184#comment-14227184 ] Brock Noland commented on HIVE-8836: Thank you [~chengxiang li]! I will apply that on the current patch. I've also created HIVE-8983 to have PTest backup the spark.log file as part of the testing process. Sync'ed with [~lirui] offline and he'll be looking at some failures. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227201#comment-14227201 ] Brock Noland commented on HIVE-8836: FYI I am generating a new patch which incorporates Jimmy's changes in HIVE-8978. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch, HIVE-8836.9-spark.patch, additional-enable-spark-log.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227316#comment-14227316
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12683990/HIVE-8836.10-spark.patch
{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 7180 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_transform
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_script_env_var1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_script_env_var2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_script_pipe
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_counter
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_transform1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_transform2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union23
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/456/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/456/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-456/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12683990 - PreCommit-HIVE-SPARK-Build
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.10-spark.patch,
HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch,
HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.7-spark.patch, HIVE-8836.8-spark.patch,
HIVE-8836.9-spark.patch, additional-enable-spark-log.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225049#comment-14225049 ] Marcelo Vanzin commented on HIVE-8836: -- I talked briefly with Brock about this, but the main thing here is that, right now, Spark is not very friendly to applications that are trying to embed it. As you've noticed, the assembly jar, which contains almost everything you need to run Spark, is not published in maven or anywhere. And not all artifacts used to build the assembly are published - for example, the Yarn backend cannot be found anywhere in maven, so without the assembly you cannot submit jobs to Yarn. I've suggested it in the past, but I think right now, or until Spark makes itself more friendly to such use cases, Hive should require a full Spark install to work. If desired we could use the hacks I added to the remote client to not need the full install for unit tests, but even those are very limited; it probably only works with a local master as some of you may have noticed. Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch, HIVE-8836-brock-3.patch, HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225114#comment-14225114 ] Brock Noland commented on HIVE-8836: I think it makes sense to move forward with the tarball approach since it works today. Then we can file a follow-on jira to come up with a cleaner approach. That approach might need some enhancements to Spark like publishing the assembly to Maven. Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch, HIVE-8836-brock-3.patch, HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225336#comment-14225336 ] Brock Noland commented on HIVE-8836: Tests of this patch are in progress here: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/433/console Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225407#comment-14225407 ] Xuefu Zhang commented on HIVE-8836: --- It seems the test is stuck. On Tue, Nov 25, 2014 at 2:45 PM, Brock Noland (JIRA) j...@apache.org Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225412#comment-14225412
]
Brock Noland commented on HIVE-8836:
Yes it looks like two commands are taking a while. Looking into it.
{noformat}
501 7997 0.0 0.0 60236 3804 ?S17:31 0:00 \_ ssh -v -i
/home/hiveptest/.ssh/hive-ptest-user-key -l hiveptest 50.18.64.184 bash
/home/hiveptest//50.18.64.184-hiveptest-1/scratch/hiveptest-TestSparkCliDriver-join11.q-join18.q-groupby2.q-and-12-more.sh
501 8211 0.0 0.0 60236 3804 ?S17:36 0:00 \_ ssh -v -i
/home/hiveptest/.ssh/hive-ptest-user-key -l hiveptest 50.18.64.184 bash
/home/hiveptest//50.18.64.184-hiveptest-2/scratch/hiveptest-TestSparkCliDriver-union_remove_9.q-ppd_multi_insert.q-auto_sortmerge_join_16.q-and-12-more.sh
{noformat}
Enable automatic tests with remote spark client.[Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225417#comment-14225417
]
Brock Noland commented on HIVE-8836:
One of the stuck processes:
{noformat}
2014-11-25 14:30:44,874 INFO ql.Driver (SessionState.java:printInfo(828)) -
Query ID = hiveptest_20141125143030_c56b5f72-8552-4122-930d-7df9dea96638
2014-11-25 14:30:44,874 INFO ql.Driver (SessionState.java:printInfo(828)) -
Total jobs = 1
2014-11-25 14:30:44,874 INFO ql.Driver (SessionState.java:printInfo(828)) -
Launching Job 1 out of 1
2014-11-25 14:30:44,875 INFO ql.Driver (Driver.java:launchTask(1643)) -
Starting task [Stage-1:MAPRED] in serial mode
2014-11-25 14:30:44,875 INFO exec.Task (SessionState.java:printInfo(828)) - In
order to change the average load for a reducer (in bytes):
2014-11-25 14:30:44,876 INFO exec.Task (SessionState.java:printInfo(828)) -
set hive.exec.reducers.bytes.per.reducer=number
2014-11-25 14:30:44,876 INFO exec.Task (SessionState.java:printInfo(828)) - In
order to limit the maximum number of reducers:
2014-11-25 14:30:44,876 INFO exec.Task (SessionState.java:printInfo(828)) -
set hive.exec.reducers.max=number
2014-11-25 14:30:44,876 INFO exec.Task (SessionState.java:printInfo(828)) - In
order to set a constant number of reducers:
2014-11-25 14:30:44,876 INFO exec.Task (SessionState.java:printInfo(828)) -
set mapreduce.job.reduces=number
2014-11-25 14:30:44,876 INFO spark.HiveSparkClientFactory
(HiveSparkClientFactory.java:initiateSparkConf(105)) - load spark configuration
from hive configuration (spark.master - local-cluster[2,1,2048]).
2014-11-25 14:30:44,894 INFO slf4j.Slf4jLogger
(Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
2014-11-25 14:30:44,899 INFO Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) -
Starting remoting
2014-11-25 14:30:44,907 INFO Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) -
Remoting started; listening on addresses
:[akka.tcp://d84d934e-e10e-4744-b42c-ed86a49ebbd3@10.227.4.181:56697]
2014-11-25 14:30:44,909 DEBUG client.SparkClientImpl
(SparkClientImpl.java:startDriver(252)) - Running client driver with argv:
/home/hiveptest/50.18.64.184-hiveptest-1/apache-svn-spark-source/itests/qtest-spark/../../itests/qtest-spark/target/spark/bin/spark-submit
--properties-file
/home/hiveptest/50.18.64.184-hiveptest-1/apache-svn-spark-source/itests/qtest-spark/target/tmp/spark-submit.8721943354350626566.properties
--class org.apache.hive.spark.client.RemoteDriver
/home/hiveptest/50.18.64.184-hiveptest-1/maven/org/apache/hive/hive-exec/0.15.0-SNAPSHOT/hive-exec-0.15.0-SNAPSHOT.jar
--remote
akka.tcp://d84d934e-e10e-4744-b42c-ed86a49ebbd3@10.227.4.181:56697/user/SparkClient-aa5b3525-a031-41d7-ab04-8a18b0aa3fcf
2014-11-25 14:30:48,058 INFO client.SparkClientImpl
(SparkClientImpl.java:onReceive(312)) - Received hello from
akka.tcp://92e75da1-125e-4576-b63e-dc3166653dbe@10.227.4.181:35948/user/RemoteDriver
2014-11-25 14:30:48,059 DEBUG session.SparkSessionManagerImpl
(SparkSessionManagerImpl.java:getSession(126)) - New session
(1743e7e0-1fb3-4766-9249-cc138f88a2a7) is created.
2014-11-25 14:30:48,085 INFO ql.Context (Context.java:getMRScratchDir(266)) -
New scratch dir is
file:/home/hiveptest/50.18.64.184-hiveptest-1/apache-svn-spark-source/itests/qtest-spark/target/tmp/scratchdir/hiveptest/cf1f8ff8-a03d-413c-9922-d7c5c3c25e18/hive_2014-11-25_14-30-44_827_2636220880717371319-1
2014-11-25 14:30:48,283 INFO client.SparkClientImpl
(SparkClientImpl.java:onReceive(329)) - Received result for
fa96fc57-3999-4103-9bbd-a4bab346a324
2014-11-25 14:30:48,535 INFO client.SparkClientImpl
(SparkClientImpl.java:onReceive(329)) - Received result for
00716575-c666-4b2a-bffe-682780684df8
2014-11-25 14:47:28,067 INFO transport.ProtocolStateActor
(Slf4jLogger.scala:apply$mcV$sp(74)) - No response from remote. Handshake timed
out or transport failure detector triggered.
2014-11-25 14:47:28,071 WARN remote.ReliableDeliverySupervisor
(Slf4jLogger.scala:apply$mcV$sp(71)) - Association with remote system
[akka.tcp://92e75da1-125e-4576-b63e-dc3166653dbe@10.227.4.181:35948] has
failed, address is now gated for [5000] ms. Reason is: [Disassociated].
{noformat}
Enable automatic tests with remote spark client.[Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch
In real production environment, remote spark client should be used to submit
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225423#comment-14225423
]
Brock Noland commented on HIVE-8836:
Similar warning on the other:
{noformat}
2014-11-25 14:39:26,714 INFO ql.Driver (SessionState.java:printInfo(828)) -
Query ID = hiveptest_20141125143939_77f4ba53-fc27-4efe-ad3d-85e23ea54748
2014-11-25 14:39:26,714 INFO ql.Driver (SessionState.java:printInfo(828)) -
Total jobs = 2
2014-11-25 14:39:26,714 INFO ql.Driver (SessionState.java:printInfo(828)) -
Launching Job 1 out of 2
2014-11-25 14:39:26,715 INFO ql.Driver (Driver.java:launchTask(1643)) -
Starting task [Stage-3:MAPRED] in serial mode
2014-11-25 14:39:26,715 INFO exec.Task (SessionState.java:printInfo(828)) - In
order to change the average load for a reducer (in bytes):
2014-11-25 14:39:26,715 INFO exec.Task (SessionState.java:printInfo(828)) -
set hive.exec.reducers.bytes.per.reducer=number
2014-11-25 14:39:26,715 INFO exec.Task (SessionState.java:printInfo(828)) - In
order to limit the maximum number of reducers:
2014-11-25 14:39:26,715 INFO exec.Task (SessionState.java:printInfo(828)) -
set hive.exec.reducers.max=number
2014-11-25 14:39:26,715 INFO exec.Task (SessionState.java:printInfo(828)) - In
order to set a constant number of reducers:
2014-11-25 14:39:26,715 INFO exec.Task (SessionState.java:printInfo(828)) -
set mapreduce.job.reduces=number
2014-11-25 14:39:26,715 DEBUG session.SparkSessionManagerImpl
(SparkSessionManagerImpl.java:getSession(107)) - Existing session
(34e37f91-2cac-4a31-aba7-85b711d8dad3) is reused.
2014-11-25 14:39:26,728 INFO ql.Context (Context.java:getMRScratchDir(266)) -
New scratch dir is
file:/home/hiveptest/50.18.64.184-hiveptest-2/apache-svn-spark-source/itests/qtest-spark/target/tmp/scratchdir/hiveptest/e9216630-f66e-4b31-bc30-58078678a976/hive_2014-11-25_14-39-26_638_5903751068675432492-1
2014-11-25 14:39:26,775 INFO client.SparkClientImpl
(SparkClientImpl.java:onReceive(329)) - Received result for
e288f226-2429-469a-9c53-07fabda12db3
2014-11-25 14:55:54,771 WARN remote.ReliableDeliverySupervisor
(Slf4jLogger.scala:apply$mcV$sp(71)) - Association with remote system
[akka.tcp://8fe8195c-f1be-45e3-a9d1-b94b7caafcd9@10.227.4.181:38320] has
failed, address is now gated for [5000] ms. Reason is: [Disassociated].
{noformat}
Enable automatic tests with remote spark client.[Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225429#comment-14225429 ] Brock Noland commented on HIVE-8836: Complete log: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/akka-error/hive.log.txt Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225438#comment-14225438 ] Brock Noland commented on HIVE-8836: Bunch of debug logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/akka-error/ Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225492#comment-14225492
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12683650/HIVE-8836.4-spark.patch
{color:red}ERROR:{color} -1 due to 96 failed/errored test(s), 7148 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_complex_types_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_rollup1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_having
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join38
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join40
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_reorder
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_filter_on_outerjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_move_tasks_share_dependencies
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_outer_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_transform
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_decimal
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_general_queries
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_semijoin
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225535#comment-14225535
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12683721/HIVE-8836.5-spark.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/437/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/437/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-437/
Messages:
{noformat}
This message was trimmed, see log for full details
Reverted 'ql/src/test/results/clientpositive/spark/load_dyn_part10.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/stats5.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/ppd_join3.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/skewjoin.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/mapjoin_distinct.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join12.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/smb_mapjoin_16.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/input17.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/ppd_join_filter.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join35.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/groupby1.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_cond_pushdown_4.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/bucketmapjoin2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/mapjoin1.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/insert_into1.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/skewjoinopt16.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/smb_mapjoin9.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/groupby11.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/column_access_stats.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join21.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_merging.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_map_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/stats0.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/smb_mapjoin_2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join9.q.out'
Reverted
'ql/src/test/results/clientpositive/spark/skewjoin_union_remove_1.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/input12.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join30.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_join6.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/mapjoin_hook.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union_remove_19.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/mapreduce2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/index_auto_self_join.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_nullsafe.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union_remove_6.q.out'
Reverted
'ql/src/test/results/clientpositive/spark/join_merge_multi_expressions.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/vector_decimal_mapjoin.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_join16.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/smb_mapjoin_20.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/bucket3.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/ppd_outer_join5.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/semijoin.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/stats_partscan_1_23.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/skewjoinopt20.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/merge2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join4.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/bucket_map_join_tez2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_join25.q.out'
Reverted
'ql/src/test/results/clientpositive/spark/join_cond_pushdown_unqual4.q.out'
Reverted
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225578#comment-14225578
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12683726/HIVE-8836.6-spark.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/438/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/438/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-438/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: SSHExecutionException: SSHResult [command=pkill -f java,
getExitCode()=1, getException()=null, getUser()=hiveptest,
getHost()=54.176.195.138, getInstance()=0]: 'OpenSSH_5.3p1, OpenSSL 1.0.0-fips
29 Mar 2010
debug1: Reading configuration data /home/hiveptest/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 54.176.195.138 [54.176.195.138] port 22.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: identity file /home/hiveptest/.ssh/hive-ptest-user-key type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server-client aes128-ctr hmac-md5 none
debug1: kex: client-server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(102410248192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '54.176.195.138' is known and matches the RSA host key.
debug1: Found key in /home/hiveptest/.ssh/known_hosts:1194
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
Address 54.176.195.138 maps to
ec2-54-176-195-138.us-west-1.compute.amazonaws.com, but this does not map back
to the address - POSSIBLE BREAK-IN ATTEMPT!
debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file '/tmp/krb5cc_501' not found
debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file '/tmp/krb5cc_501' not found
debug1: Unspecified GSS failure. Minor code may provide more information
debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file '/tmp/krb5cc_501' not found
debug1: Next authentication method: publickey
debug1: Trying private key: /home/hiveptest/.ssh/hive-ptest-user-key
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessi...@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: pkill -f java
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype e...@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 1952, received 1976 bytes, in 0.0 seconds
Bytes per second: sent 59126.3, received 59853.3
debug1: Exit status 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12683726 - PreCommit-HIVE-SPARK-Build
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225716#comment-14225716
]
Brock Noland commented on HIVE-8836:
Again with the latest patch two tests are stuck. I copied all relavent info to:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/akka-error/2/
if you look at the end of the hive.log:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/akka-error/2/qtest-spark/target/tmp/log/
it seems to get into a state where {{SparkJobMonitor}} is only getting an
{{UKNOWN}} state from {{SparkJobStatus}}
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225721#comment-14225721 ] Rui Li commented on HIVE-8836: -- Hi [~brocknoland], this may be related to HIVE-8956. Do you know which two tests are stuck? Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225730#comment-14225730
]
Brock Noland commented on HIVE-8836:
Looks like it's:
{noformat}
Begin query: optimize_nullscan.q
Begin query: custom_input_output_format.q
{noformat}
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225732#comment-14225732 ] Brock Noland commented on HIVE-8836: [~lirui] that does look like it might be related! Nice find. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225742#comment-14225742
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12683735/HIVE-8836.6-spark.patch
{color:red}ERROR:{color} -1 due to 105 failed/errored test(s), 7148 tests
executed
*Failed tests:*
{noformat}
TestAuthorizationApiAuthorizer - did not produce a TEST-*.xml file
TestGenericUDFOPNumeric - did not produce a TEST-*.xml file
TestHBaseKeyFactory - did not produce a TEST-*.xml file
TestHBaseKeyFactory2 - did not produce a TEST-*.xml file
TestHBaseKeyFactory3 - did not produce a TEST-*.xml file
TestHBasePredicateDecomposer - did not produce a TEST-*.xml file
TestSparkCliDriver-join11.q-join18.q-groupby2.q-and-12-more - did not produce a
TEST-*.xml file
TestSparkCliDriver-union_remove_9.q-ppd_multi_insert.q-auto_sortmerge_join_16.q-and-12-more
- did not produce a TEST-*.xml file
TestTezSessionState - did not produce a TEST-*.xml file
TestURLHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_complex_types_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_rollup1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_having
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join38
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join40
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_reorder
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_filter_on_outerjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_move_tasks_share_dependencies
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_outer_join_ppr
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225751#comment-14225751 ] Rui Li commented on HIVE-8836: -- OK I'll try out the failed tests. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225754#comment-14225754
]
Brock Noland commented on HIVE-8836:
{noformat}
TestSparkCliDriver-join11.q-join18.q-groupby2.q-and-12-more - did not produce a
TEST-*.xml file
TestSparkCliDriver-union_remove_9.q-ppd_multi_insert.q-auto_sortmerge_join_16.q-and-12-more
- did not produce a TEST-*.xml file
{noformat}
The two tests above were the hung tests. Other did not produce a ... items
will be resolved by HIVE-8971. We'll commit HIVE-8956 to fix the hangs.
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225779#comment-14225779
]
Xuefu Zhang commented on HIVE-8836:
---
Many test failures above are shown as plan diff, somehting like:
{code}
211c211
Reducer 2 - Map 1 (GROUP, 3)
---
Reducer 2 - Map 1 (GROUP, 1)
{code}
This can be explained by the fact that the current algorithm determining the
number of reducers has included the factors such as number of executors in the
cluster. Because we switched to local-cluster with two nodes, the number of
reducers is different from previous value. This is not good, because every time
we change the testing cluster, we might have a different plan.
Enable automatic tests with remote spark client [Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch,
HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225795#comment-14225795 ] Brock Noland commented on HIVE-8836: bq. This is not good, because every time we change the testing cluster, we might have a different plan. How often do you think we'll be changing the local cluster size? My guess is not that often? Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225803#comment-14225803 ] Xuefu Zhang commented on HIVE-8836: --- Yeah. but it also means that when dev generates the .out file, same cluster configuration is required. I guess we can live with that for now. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225849#comment-14225849
]
Hive QA commented on HIVE-8836:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12683770/HIVE-8836.6-spark.patch
{color:red}ERROR:{color} -1 due to 108 failed/errored test(s), 7178 tests
executed
*Failed tests:*
{noformat}
TestAuthorizationApiAuthorizer - did not produce a TEST-*.xml file
TestGenericUDFOPNumeric - did not produce a TEST-*.xml file
TestHBaseKeyFactory - did not produce a TEST-*.xml file
TestHBaseKeyFactory2 - did not produce a TEST-*.xml file
TestHBaseKeyFactory3 - did not produce a TEST-*.xml file
TestHBasePredicateDecomposer - did not produce a TEST-*.xml file
TestTezSessionState - did not produce a TEST-*.xml file
TestURLHook - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_acid
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_complex_types_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_rollup1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_having
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join38
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join40
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_reorder
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_filter_on_outerjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_move_tasks_share_dependencies
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_outer_join_ppr
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225860#comment-14225860 ] Rui Li commented on HIVE-8836: -- Shall we use same #reducers as before for now? So that we can tell which tests are actually failing and which are just affected by the #reducers change. Enable automatic tests with remote spark client [Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch, HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224065#comment-14224065 ] Chengxiang Li commented on HIVE-8836: - The join0.q output diff should because of HIVE-8855 which jimmy is working on. Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch, HIVE-8836-brock-3.patch, HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224098#comment-14224098
]
Szehon Ho commented on HIVE-8836:
-
bq. Spark assembly dependency can be fetched from public maven reposiotry
Hi Chengxiang, I dont understand this part, how are you downloading this from
public repo? When I try your patch I get the error:
{noformat}
Could not resolve dependencies for project
org.apache.hive:hive-it-qfile-spark:jar:0.15.0-SNAPSHOT: Could not find
artifact org.apache.spark:spark-assembly_2.10:jar:1.2.0-SNAPSHOT in
spark-snapshot
(http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/
{noformat}
And we cant find it in any public repo. Thats why we assumed we had to build
it and upload to hosted location.
Also another question, as we were trying to set spark.home, which looks for
bin/spark-submit, which then pulled in scripts like compute-classpath.sh,
load-spark-env.sh, spark-class, and finally spark-assembly itself. I see you
are using another way (spark.test.home, spark.testing), how does that avoid
looking for these artifacts to start the spark process?
Enable automatic tests with remote spark client.[Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch,
HIVE-8836-brock-3.patch, HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224100#comment-14224100 ] Brock Noland commented on HIVE-8836: bq. Spark assembly dependency can be fetched from public maven reposiotry, it should be more maintainable than download a whole spark installation. what do you think? I don't think spark assembly is published to maven. The Sentry project has used the download a tarball (of Hive and Hadoop) for over a year now and it works well. However, if there is a lightweight method of making this work, I am all for it. Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch, HIVE-8836-brock-3.patch, HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224124#comment-14224124
]
Chengxiang Li commented on HIVE-8836:
-
Hi, [~szehon] and [~brocknoland]
I'm not 100% percent sure spark assembly jar would be published to public maven
repository, but I find spark assembly at
[here|http://mvnrepository.com/artifact/org.apache.spark/spark-assembly_2.10/1.1.0],
maybe [~vanzin] know more about this. There is no
org.apache.spark:spark-assembly_2.10:jar:1.2.0-SNAPSHOT in any public maven
repository yet as it's still in SNAPSHOT status, but we can publish it to
http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data as what we have
done for spark core. I build spark and public to local maven repository in my
local test.
{quote}
Also another question, as we were trying to set spark.home, which looks for
bin/spark-submit, which then pulled in scripts like compute-classpath.sh,
load-spark-env.sh, spark-class, and finally spark-assembly itself. I see you
are using another way (spark.test.home, spark.testing), how does that avoid
looking for these artifacts to start the spark process?
{quote}
First, bin/spark-submit is optional for Remote Spark Context.
Then, local-cluster spark only need compute-classpath.sh for launch executor,
which is used to add spark related jars into classpath(Hive unit test should
only need spark-assembly). spark.test.home and spark.testing are used to set
spark home to dummy spark installation, you can check
org.apache.spark.deploy.worker.Worker::line101 for why. I create dummy spark
installation with empty compute-classpath.sh as compute-classpath.sh is
required, and add spark assembly to spark executor classpath through
spark.executor.extraClassPath.
Enable automatic tests with remote spark client.[Spark Branch]
--
Key: HIVE-8836
URL: https://issues.apache.org/jira/browse/HIVE-8836
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch,
HIVE-8836-brock-3.patch, HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch
In real production environment, remote spark client should be used to submit
spark job for Hive mostly, we should enable automatic test with remote spark
client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224170#comment-14224170 ] Szehon Ho commented on HIVE-8836: - Thanks for the details, Chengxiang. Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch, HIVE-8836-brock-3.patch, HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208088#comment-14208088 ] Xuefu Zhang commented on HIVE-8836: --- Hi [~chengxiang li], With HIVE-8795, this should get enabled, right? I understand that we need to verify this. Thanks. Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Labels: Spark-M3 In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209035#comment-14209035 ] Chengxiang Li commented on HIVE-8836: - Yes, [~xuefuz], this would be enabled with HIVE-8795 and HIVE-8833, I suspect several tests would failed, this JIRA is listed here to notify us to track the failed tests. Enable automatic tests with remote spark client.[Spark Branch] -- Key: HIVE-8836 URL: https://issues.apache.org/jira/browse/HIVE-8836 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Labels: Spark-M3 In real production environment, remote spark client should be used to submit spark job for Hive mostly, we should enable automatic test with remote spark client to make sure the Hive feature workable with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
