[jira] [Commented] (HIVE-9059) Remove wrappers for SparkJobInfo and SparkStageInfo
[
https://issues.apache.org/jira/browse/HIVE-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245249#comment-14245249
]
Hive QA commented on HIVE-9059:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12686183/HIVE-9059.1-spark.patch
{color:red}ERROR:{color} -1 due to 408 failed/errored test(s), 6706 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join30.q-sort_merge_join_desc_3.q-join2.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-avro_joins.q-join36.q-join4.q-and-12-more - did not produce
a TEST-*.xml file
TestSparkCliDriver-bucketmapjoin_negative.q-parallel_join0.q-insert_into1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-skewjoin_noskew.q-mapreduce1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby10.q-union5.q-load_dyn_part15.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-skewjoinopt8.q-union_remove_1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-groupby4.q-tez_joins_explain.q-load_dyn_part3.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby_complex_types.q-auto_join9.q-groupby_map_ppr.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-vectorization_16.q-multi_insert_mixed.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-input1_limit.q-groupby8_map.q-varchar_join1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-join11.q-join18.q-groupby2.q-and-12-more - did not produce a
TEST-*.xml file
TestSparkCliDriver-join13.q-sample2.q-join_reorder3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-join39.q-bucketmapjoin10.q-bucket_map_join_tez1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join9.q-ppd_outer_join2.q-join_casesensitive.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_3.q-groupby7.q-auto_join17.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_unqual4.q-bucketmapjoin12.q-avro_decimal_native.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_filters.q-bucketmapjoin6.q-smb_mapjoin_4.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-list_bucket_dml_2.q-multi_insert.q-join5.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-load_dyn_part5.q-load_dyn_part2.q-auto_join10.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join1.q-escape_distributeby1.q-timestamp_null.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-skewjoinopt3.q-auto_join1.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-ppd_transform.q-auto_sortmerge_join_7.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ptf_general_queries.q-bucketmapjoin3.q-enforce_order.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ptf_seqfile.q-auto_join18.q-vectorization_10.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoin_union_remove_2.q-auto_join14.q-auto_join20.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt15.q-union_remove_7.q-avro_joins_native.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-smb_mapjoin_15.q-auto_join18_multi_distinct.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-sort_merge_join_desc_1.q-order.q-union2.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-stats12.q-auto_join24.q-vector_decimal_aggregate.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-stats13.q-stats2.q-ppd_gby_join.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-table_access_keys_stats.q-bucketsortoptimize_insert_4.q-union_remove_9.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-auto_join_reordering_values.q-union29.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-transform_ppr2.q-join20.q-decimal_join.q-and-11-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-vector_distinct_2.q-join15.q-union19.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-bucket3.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hadoop.hive.ql.exec.TestExecDriver.initializationError
[jira] [Commented] (HIVE-9091) Add additional unit tests for HiveSQLException
[
https://issues.apache.org/jira/browse/HIVE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245255#comment-14245255
]
Hive QA commented on HIVE-9091:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12686938/HIVE-9091.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6708 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2069/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2069/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2069/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12686938 - PreCommit-HIVE-TRUNK-Build
Add additional unit tests for HiveSQLException
--
Key: HIVE-9091
URL: https://issues.apache.org/jira/browse/HIVE-9091
Project: Hive
Issue Type: Test
Components: JDBC
Reporter: Aihua Xu
Assignee: Aihua Xu
Priority: Minor
Labels: easyfix, test
Fix For: 0.15.0
Attachments: HIVE-9091.1.patch
Original Estimate: 12h
Remaining Estimate: 12h
Change the function scope for the functions in HiveSQLException to be
private, including
toString(Throwable cause, StackTraceElement[] parent) ,
enroll(Throwable ex, StackTraceElement[] trace, int max),
toStackTrace(ListString details, StackTraceElement[] parent, int index)
since these serve as helper function internally.
Add comments for the public functions in HiveSQLException.
To add additional unit test cases for HiveSQLException such as exception with
unknown source, test TStatus conversion, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8911) Enable mapjoin hints [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245259#comment-14245259
]
Hive QA commented on HIVE-8911:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12686982/HIVE-8911.6-spark.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 7233 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_25
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/534/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/534/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-534/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12686982 - PreCommit-HIVE-SPARK-Build
Enable mapjoin hints [Spark Branch]
---
Key: HIVE-8911
URL: https://issues.apache.org/jira/browse/HIVE-8911
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Szehon Ho
Assignee: Chao
Attachments: HIVE-8911.1-spark.patch, HIVE-8911.2-spark.patch,
HIVE-8911.3-spark.patch, HIVE-8911.4-spark.patch, HIVE-8911.5-spark.patch,
HIVE-8911.6-spark.patch
Currently the big table selection in a mapjoin is based on stats.
We should also enable the big-table selection based on hints. See class
MapJoinProcessor. This is a logical-optimizer class, so we should be able to
re-use this without too many changes to hook up with SparkMapJoinResolver.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8866) Vectorization on partitioned table throws ArrayIndexOutOfBoundsException when partitions are not of same #of columns
[
https://issues.apache.org/jira/browse/HIVE-8866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245265#comment-14245265
]
Hive QA commented on HIVE-8866:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12686946/HIVE-8866.08.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6704 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_covar_pop
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2070/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2070/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2070/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12686946 - PreCommit-HIVE-TRUNK-Build
Vectorization on partitioned table throws ArrayIndexOutOfBoundsException when
partitions are not of same #of columns
Key: HIVE-8866
URL: https://issues.apache.org/jira/browse/HIVE-8866
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.1
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Fix For: 0.14.1
Attachments: HIVE-8866.01.patch, HIVE-8866.02.patch,
HIVE-8866.03.patch, HIVE-8866.05.patch, HIVE-8866.06.patch,
HIVE-8866.07.patch, HIVE-8866.08.patch
Vectorization assumes partitions are of same number of columns, and takes
upon # of columns on first read. consequent addPartitionColsToBatch throws
ArrayIndexOutOfboundsException if the # columns is bigger.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9084) Investigate IOContext object initialization problem [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245281#comment-14245281
]
Hive QA commented on HIVE-9084:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12687021/HIVE-9084.2-spark.patch
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 7261 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_virtual_column
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.ql.exec.TestOperators.testMapOperator
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/535/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/535/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-535/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12687021 - PreCommit-HIVE-SPARK-Build
Investigate IOContext object initialization problem [Spark Branch]
--
Key: HIVE-9084
URL: https://issues.apache.org/jira/browse/HIVE-9084
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-9084.1-spark.patch, HIVE-9084.2-spark.patch
In recent ptest run (Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/511/testReport),
test groupby_multi_single_reducer.q failed w/ the following stacktrace:
{code}
java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:136)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:54)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:29)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:167)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:167)
at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:601)
at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:601)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.io.IOContext.copy(IOContext.java:119)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:97)
... 16 more
{code}
This failure is again about IOContext object, which needs further
investigation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8937) fix description of hive.security.authorization.sqlstd.confwhitelist.* params
[ https://issues.apache.org/jira/browse/HIVE-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245284#comment-14245284 ] Lefty Leverenz commented on HIVE-8937: -- Should I document *hive.security.authorization.sqlstd.confwhitelist.append* with its original description (HIVE-8534) or wait for this revision? fix description of hive.security.authorization.sqlstd.confwhitelist.* params Key: HIVE-8937 URL: https://issues.apache.org/jira/browse/HIVE-8937 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Thejas M Nair Assignee: Thejas M Nair hive.security.authorization.sqlstd.confwhitelist.* param description in HiveConf is incorrect. The expected value is a regex, not comma separated regexes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8534) sql std auth : update configuration whitelist for 0.14
[ https://issues.apache.org/jira/browse/HIVE-8534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245285#comment-14245285 ] Lefty Leverenz commented on HIVE-8534: -- Partially documented: * [Configuration Properties -- hive.security.authorization.sqlstd.confwhitelist | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.security.authorization.sqlstd.confwhitelist] sql std auth : update configuration whitelist for 0.14 -- Key: HIVE-8534 URL: https://issues.apache.org/jira/browse/HIVE-8534 Project: Hive Issue Type: Bug Components: Authorization, SQLStandardAuthorization Reporter: Thejas M Nair Assignee: Thejas M Nair Priority: Blocker Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-8534.1.patch, HIVE-8534.2.patch, HIVE-8534.3.patch, HIVE-8534.4.patch, HIVE-8534.5.patch New config parameters have been introduced in hive 0.14. SQL standard authorization needs to be updated to allow some new parameters to be set, when the authorization mode is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[
https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245286#comment-14245286
]
Hive QA commented on HIVE-8950:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12686977/HIVE-8950.5.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6715 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2071/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2071/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2071/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12686977 - PreCommit-HIVE-TRUNK-Build
Add support in ParquetHiveSerde to create table schema from a parquet file
--
Key: HIVE-8950
URL: https://issues.apache.org/jira/browse/HIVE-8950
Project: Hive
Issue Type: Improvement
Reporter: Ashish Kumar Singh
Assignee: Ashish Kumar Singh
Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch,
HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.patch
PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without
having to specify the column names and types. As, parquet files store schema
in their footer, it is possible to generate hive schema from parquet file's
metadata. This will improve usability of parquet backed tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8982) IndexOutOfBounds exception in mapjoin [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245305#comment-14245305
]
Hive QA commented on HIVE-8982:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12687022/HIVE-8982.2-spark.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 7261 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/536/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/536/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-536/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12687022 - PreCommit-HIVE-SPARK-Build
IndexOutOfBounds exception in mapjoin [Spark Branch]
Key: HIVE-8982
URL: https://issues.apache.org/jira/browse/HIVE-8982
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Szehon Ho
Assignee: Chao
Attachments: HIVE-8982.1-spark.patch, HIVE-8982.2-spark.patch
There are sometimes random failures in spark mapjoin during unit tests like:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.closeOp(SparkHashTableSinkOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:185)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1365)
at
org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1365)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinEagerRowContainer.first(MapJoinEagerRowContainer.java:70)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinEagerRowContainer.write(MapJoinEagerRowContainer.java:150)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.persist(MapJoinTableContainerSerDe.java:167)
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.flushToFile(SparkHashTableSinkOperator.java:128)
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.closeOp(SparkHashTableSinkOperator.java:77)
... 20 more
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
[jira] [Commented] (HIVE-9053) select constant in union all followed by group by gives wrong result
[
https://issues.apache.org/jira/browse/HIVE-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245318#comment-14245318
]
Hive QA commented on HIVE-9053:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12686979/HIVE-9053.03.patch
{color:green}SUCCESS:{color} +1 6703 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2072/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2072/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2072/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12686979 - PreCommit-HIVE-TRUNK-Build
select constant in union all followed by group by gives wrong result
Key: HIVE-9053
URL: https://issues.apache.org/jira/browse/HIVE-9053
Project: Hive
Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-9053.01.patch, HIVE-9053.02.patch,
HIVE-9053.03.patch
Here is the the way to reproduce with q test:
select key from (select '1' as key from src union all select key from src)tab
group by key;
will give
OK
NULL
1
This is not correct as src contains many other keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8395) CBO: enable by default
[
https://issues.apache.org/jira/browse/HIVE-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245367#comment-14245367
]
Hive QA commented on HIVE-8395:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12686988/HIVE-8395.32.patch
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 6702 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_louter_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_router_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_case_column_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join1
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2073/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2073/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2073/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12686988 - PreCommit-HIVE-TRUNK-Build
CBO: enable by default
--
Key: HIVE-8395
URL: https://issues.apache.org/jira/browse/HIVE-8395
Project: Hive
Issue Type: Improvement
Components: CBO
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: 0.15.0
Attachments: HIVE-8395-27-28-delta.patch,
HIVE-8395-28-29-delta.patch, HIVE-8395.01.patch, HIVE-8395.02.patch,
HIVE-8395.03.patch, HIVE-8395.04.patch, HIVE-8395.05.patch,
HIVE-8395.06.patch, HIVE-8395.07.patch, HIVE-8395.08.patch,
HIVE-8395.09.patch, HIVE-8395.10.patch, HIVE-8395.11.patch,
HIVE-8395.12.patch, HIVE-8395.12.patch, HIVE-8395.13.patch,
HIVE-8395.13.patch, HIVE-8395.14.patch, HIVE-8395.15.patch,
HIVE-8395.16.patch, HIVE-8395.17.patch, HIVE-8395.18.patch,
HIVE-8395.18.patch, HIVE-8395.19.patch, HIVE-8395.20.patch,
HIVE-8395.21.patch, HIVE-8395.22.patch, HIVE-8395.23.patch,
HIVE-8395.23.withon.patch, HIVE-8395.24.patch, HIVE-8395.25.patch,
HIVE-8395.25.patch, HIVE-8395.26.patch, HIVE-8395.27.patch,
HIVE-8395.28.patch, HIVE-8395.29.patch, HIVE-8395.30.patch,
HIVE-8395.31.patch, HIVE-8395.32.patch, HIVE-8395.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8982) IndexOutOfBounds exception in mapjoin [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-8982:
--
Resolution: Fixed
Fix Version/s: spark-branch
Status: Resolved (was: Patch Available)
Committed to Spark branch. Thanks to Chao for this nice finding and fix.
IndexOutOfBounds exception in mapjoin [Spark Branch]
Key: HIVE-8982
URL: https://issues.apache.org/jira/browse/HIVE-8982
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Szehon Ho
Assignee: Chao
Fix For: spark-branch
Attachments: HIVE-8982.1-spark.patch, HIVE-8982.2-spark.patch
There are sometimes random failures in spark mapjoin during unit tests like:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.closeOp(SparkHashTableSinkOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:185)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1365)
at
org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1365)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinEagerRowContainer.first(MapJoinEagerRowContainer.java:70)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinEagerRowContainer.write(MapJoinEagerRowContainer.java:150)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.persist(MapJoinTableContainerSerDe.java:167)
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.flushToFile(SparkHashTableSinkOperator.java:128)
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.closeOp(SparkHashTableSinkOperator.java:77)
... 20 more
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.closeOp(SparkHashTableSinkOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:185)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
[jira] [Commented] (HIVE-8911) Enable mapjoin hints [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245386#comment-14245386 ] Xuefu Zhang commented on HIVE-8911: --- ppd_join4 and smb_mapjoin_25 are not related, but bucketmapjoin10 might be. Enable mapjoin hints [Spark Branch] --- Key: HIVE-8911 URL: https://issues.apache.org/jira/browse/HIVE-8911 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Chao Attachments: HIVE-8911.1-spark.patch, HIVE-8911.2-spark.patch, HIVE-8911.3-spark.patch, HIVE-8911.4-spark.patch, HIVE-8911.5-spark.patch, HIVE-8911.6-spark.patch Currently the big table selection in a mapjoin is based on stats. We should also enable the big-table selection based on hints. See class MapJoinProcessor. This is a logical-optimizer class, so we should be able to re-use this without too many changes to hook up with SparkMapJoinResolver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9059) Remove wrappers for SparkJobInfo and SparkStageInfo
[ https://issues.apache.org/jira/browse/HIVE-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9059: -- Attachment: HIVE-9059.1-spark.patch Remove wrappers for SparkJobInfo and SparkStageInfo --- Key: HIVE-9059 URL: https://issues.apache.org/jira/browse/HIVE-9059 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Chengxiang Li Attachments: HIVE-9059.1-spark.patch, HIVE-9059.1-spark.patch SPARK-4567 is resolved. We can remove the wrappers we added to solve the serailization issues. CLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9059) Remove wrappers for SparkJobInfo and SparkStageInfo
[ https://issues.apache.org/jira/browse/HIVE-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9059: -- Description: SPARK-4567 is resolved. We can remove the wrappers we added to solve the serailization issues. (was: SPARK-4567 is resolved. We can remove the wrappers we added to solve the serailization issues. CLEAR LIBRARY CACHE) Remove wrappers for SparkJobInfo and SparkStageInfo --- Key: HIVE-9059 URL: https://issues.apache.org/jira/browse/HIVE-9059 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Chengxiang Li Attachments: HIVE-9059.1-spark.patch, HIVE-9059.1-spark.patch SPARK-4567 is resolved. We can remove the wrappers we added to solve the serailization issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28941: HIVE-8988
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28941/ --- (Updated Dec. 13, 2014, 4:06 p.m.) Review request for hive, John Pullokkaran and Julian Hyde. Changes --- Created HiveGroupingID operator. Bugs: HIVE-8988 https://issues.apache.org/jira/browse/HIVE-8988 Repository: hive-git Description --- HIVE-8988 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveGroupingID.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java e6e6fe3a1152df82c0176bc9617ddad0ed404f0b ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java c02a65e2041e4742a56cf4a935da0a7c04d18fdb ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java 56cb4e81e16a077c5e4fc11b9d651ef8c3c241af ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 6c8624f59d9cab53fe096e76d23852c922bbbc30 ql/src/test/queries/clientpositive/groupby_cube1.q c12720b27059075050fc92d9f31420c081303699 ql/src/test/results/clientpositive/groupby_cube1.q.out 7b5d70ae8ffce47a4b351ed9dfedcd15ab1e139c Diff: https://reviews.apache.org/r/28941/diff/ Testing --- Thanks, Jesús Camacho Rodríguez
[jira] [Updated] (HIVE-8988) Support advanced aggregation in Hive to Calcite path
[ https://issues.apache.org/jira/browse/HIVE-8988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-8988: -- Attachment: HIVE-8988.02.patch [~julianhyde], thanks for the feedback. I uploaded a new patch. Support advanced aggregation in Hive to Calcite path - Key: HIVE-8988 URL: https://issues.apache.org/jira/browse/HIVE-8988 Project: Hive Issue Type: Improvement Affects Versions: 0.15.0 Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Labels: grouping, logical, optiq Fix For: 0.15.0 Attachments: HIVE-8988.01.patch, HIVE-8988.02.patch, HIVE-8988.patch CLEAR LIBRARY CACHE To close the gap between Hive and Calcite, we need to support the translation of GroupingSets into Calcite; currently this is not implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9084) Investigate IOContext object initialization problem [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9084:
--
Attachment: HIVE-9084.2-spark.patch
Investigate IOContext object initialization problem [Spark Branch]
--
Key: HIVE-9084
URL: https://issues.apache.org/jira/browse/HIVE-9084
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-9084.1-spark.patch, HIVE-9084.2-spark.patch,
HIVE-9084.2-spark.patch
In recent ptest run (Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/511/testReport),
test groupby_multi_single_reducer.q failed w/ the following stacktrace:
{code}
java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:136)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:54)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:29)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:167)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:167)
at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:601)
at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:601)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.io.IOContext.copy(IOContext.java:119)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:97)
... 16 more
{code}
This failure is again about IOContext object, which needs further
investigation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8911) Enable mapjoin hints [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245401#comment-14245401
]
Chao commented on HIVE-8911:
bucketmapjoin10 failed because of the IndexOutOfBoundException:
{noformat}
java.lang.RuntimeException: Hive Runtime Error while closing operators
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:207)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:108)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$2.apply(AsyncRDDActions.scala:115)
at
org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at
org.apache.spark.SparkContext$$anonfun$30.apply(SparkContext.scala:1390)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.closeOp(SparkHashTableSinkOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:185)
... 15 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinEagerRowContainer.first(MapJoinEagerRowContainer.java:70)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinEagerRowContainer.write(MapJoinEagerRowContainer.java:149)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.persist(MapJoinTableContainerSerDe.java:170)
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.flushToFile(SparkHashTableSinkOperator.java:142)
at
org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.closeOp(SparkHashTableSinkOperator.java:81)
... 20 more
{noformat}
This run may have not used the patch from HIVE-8982 - it is committed at
07:33am, while the run ended at 07:49am.
Enable mapjoin hints [Spark Branch]
---
Key: HIVE-8911
URL: https://issues.apache.org/jira/browse/HIVE-8911
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Szehon Ho
Assignee: Chao
Attachments: HIVE-8911.1-spark.patch, HIVE-8911.2-spark.patch,
HIVE-8911.3-spark.patch, HIVE-8911.4-spark.patch, HIVE-8911.5-spark.patch,
HIVE-8911.6-spark.patch
Currently the big table selection in a mapjoin is based on stats.
We should also enable the big-table selection based on hints. See class
MapJoinProcessor. This is a logical-optimizer class, so we should be able to
re-use this without too many changes to hook up with SparkMapJoinResolver.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8988) Support advanced aggregation in Hive to Calcite path
[
https://issues.apache.org/jira/browse/HIVE-8988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245408#comment-14245408
]
Hive QA commented on HIVE-8988:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12687044/HIVE-8988.02.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2074/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2074/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2074/
Messages:
{noformat}
This message was trimmed, see log for full details
warning(200): IdentifiersParser.g:68:4:
Decision can match input such as LPAREN LPAREN KW_FALSE using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4:
Decision can match input such as LPAREN KW_NOT SmallintLiteral using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5:
Decision can match input such as KW_CLUSTER KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5:
Decision can match input such as KW_PARTITION KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5:
Decision can match input such as KW_DISTRIBUTE KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5:
Decision can match input such as KW_SORT KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7:
Decision can match input such as STAR using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_ARRAY using multiple alternatives: 2, 6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_STRUCT using multiple alternatives: 4, 6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_UNIONTYPE using multiple alternatives: 5,
6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:270:5:
Decision can match input such as KW_TRUE using multiple alternatives: 2, 7
As a result, alternative(s) 7 were disabled for that input
warning(200): IdentifiersParser.g:270:5:
Decision can match input such as KW_FALSE using multiple alternatives: 2, 7
As a result, alternative(s) 7 were disabled for that input
warning(200): IdentifiersParser.g:270:5:
Decision can match input such as KW_NULL using multiple alternatives: 1, 7
As a result, alternative(s) 7 were disabled for that input
warning(200): IdentifiersParser.g:401:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:401:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:401:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:401:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:401:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:401:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT
KW_OVERWRITE using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:401:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION
KW_ALL using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:401:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT
KW_INTO using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that
Hive-0.14 - Build # 776 - Fixed
Changes for Build #775 Changes for Build #776 No tests ran. The Apache Jenkins build system has built Hive-0.14 (build #776) Status: Fixed Check console output at https://builds.apache.org/job/Hive-0.14/776/ to view the results.
[jira] [Updated] (HIVE-8911) Enable mapjoin hints [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-8911: -- Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Awesome! Patch committed to Spark branch. Thanks, Chao. Enable mapjoin hints [Spark Branch] --- Key: HIVE-8911 URL: https://issues.apache.org/jira/browse/HIVE-8911 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Chao Fix For: spark-branch Attachments: HIVE-8911.1-spark.patch, HIVE-8911.2-spark.patch, HIVE-8911.3-spark.patch, HIVE-8911.4-spark.patch, HIVE-8911.5-spark.patch, HIVE-8911.6-spark.patch Currently the big table selection in a mapjoin is based on stats. We should also enable the big-table selection based on hints. See class MapJoinProcessor. This is a logical-optimizer class, so we should be able to re-use this without too many changes to hook up with SparkMapJoinResolver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9053) select constant in union all followed by group by gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245445#comment-14245445 ] Pengcheng Xiong commented on HIVE-9053: --- Good to hear that all tests passed. I still would like to investigate another small corner issue before checked in... Thanks! select constant in union all followed by group by gives wrong result Key: HIVE-9053 URL: https://issues.apache.org/jira/browse/HIVE-9053 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9053.01.patch, HIVE-9053.02.patch, HIVE-9053.03.patch Here is the the way to reproduce with q test: select key from (select '1' as key from src union all select key from src)tab group by key; will give OK NULL 1 This is not correct as src contains many other keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9059) Remove wrappers for SparkJobInfo and SparkStageInfo
[ https://issues.apache.org/jira/browse/HIVE-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245495#comment-14245495 ] Xuefu Zhang commented on HIVE-9059: --- [~chengxiang li], the patch seems making the ptest hang for some reason. Remove wrappers for SparkJobInfo and SparkStageInfo --- Key: HIVE-9059 URL: https://issues.apache.org/jira/browse/HIVE-9059 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Chengxiang Li Attachments: HIVE-9059.1-spark.patch, HIVE-9059.1-spark.patch SPARK-4567 is resolved. We can remove the wrappers we added to solve the serailization issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245509#comment-14245509 ] Ashish Kumar Singh commented on HIVE-8950: -- The test failure above does not look related to this patch. Add support in ParquetHiveSerde to create table schema from a parquet file -- Key: HIVE-8950 URL: https://issues.apache.org/jira/browse/HIVE-8950 Project: Hive Issue Type: Improvement Reporter: Ashish Kumar Singh Assignee: Ashish Kumar Singh Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.patch PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without having to specify the column names and types. As, parquet files store schema in their footer, it is possible to generate hive schema from parquet file's metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9059) Remove wrappers for SparkJobInfo and SparkStageInfo
[
https://issues.apache.org/jira/browse/HIVE-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245529#comment-14245529
]
Hive QA commented on HIVE-9059:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12687043/HIVE-9059.1-spark.patch
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 6731 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join30.q-sort_merge_join_desc_3.q-join2.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-avro_joins.q-join36.q-join4.q-and-12-more - did not produce
a TEST-*.xml file
TestSparkCliDriver-bucketmapjoin_negative.q-parallel_join0.q-insert_into1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-skewjoin_noskew.q-mapreduce1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby10.q-union5.q-load_dyn_part15.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-skewjoinopt8.q-union_remove_1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-groupby4.q-tez_joins_explain.q-load_dyn_part3.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby_complex_types.q-auto_join9.q-groupby_map_ppr.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-vectorization_16.q-multi_insert_mixed.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-input1_limit.q-groupby8_map.q-varchar_join1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-join11.q-join18.q-groupby2.q-and-12-more - did not produce a
TEST-*.xml file
TestSparkCliDriver-join13.q-sample2.q-join_reorder3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-join39.q-bucketmapjoin10.q-bucket_map_join_tez1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join9.q-ppd_outer_join2.q-join_casesensitive.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_3.q-groupby7.q-auto_join17.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_unqual4.q-bucketmapjoin12.q-avro_decimal_native.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_filters.q-bucketmapjoin6.q-smb_mapjoin_4.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-list_bucket_dml_2.q-multi_insert.q-join5.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-load_dyn_part5.q-load_dyn_part2.q-auto_join10.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join1.q-escape_distributeby1.q-timestamp_null.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-skewjoinopt3.q-auto_join1.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-ppd_transform.q-auto_sortmerge_join_7.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ptf_general_queries.q-bucketmapjoin3.q-enforce_order.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ptf_seqfile.q-auto_join18.q-vectorization_10.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoin_union_remove_2.q-auto_join14.q-auto_join20.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt15.q-union_remove_7.q-avro_joins_native.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-smb_mapjoin_15.q-auto_join18_multi_distinct.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-sort_merge_join_desc_1.q-order.q-union2.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-stats12.q-auto_join24.q-vector_decimal_aggregate.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-stats13.q-stats2.q-ppd_gby_join.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-table_access_keys_stats.q-bucketsortoptimize_insert_4.q-union_remove_9.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-auto_join_reordering_values.q-union29.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-transform_ppr2.q-join20.q-decimal_join.q-and-11-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-vector_distinct_2.q-join15.q-union19.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-bucket3.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
{noformat}
Test results:
[jira] [Commented] (HIVE-9084) Investigate IOContext object initialization problem [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245582#comment-14245582
]
Hive QA commented on HIVE-9084:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12687045/HIVE-9084.2-spark.patch
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 7233 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_virtual_column
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.ql.exec.TestOperators.testMapOperator
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/538/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/538/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-538/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12687045 - PreCommit-HIVE-SPARK-Build
Investigate IOContext object initialization problem [Spark Branch]
--
Key: HIVE-9084
URL: https://issues.apache.org/jira/browse/HIVE-9084
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-9084.1-spark.patch, HIVE-9084.2-spark.patch,
HIVE-9084.2-spark.patch
In recent ptest run (Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/511/testReport),
test groupby_multi_single_reducer.q failed w/ the following stacktrace:
{code}
java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:136)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:54)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:29)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:167)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:167)
at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:601)
at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:601)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.io.IOContext.copy(IOContext.java:119)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:97)
... 16 more
{code}
This failure is again about IOContext object, which needs further
investigation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8911) Enable mapjoin hints [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245668#comment-14245668 ] Lefty Leverenz commented on HIVE-8911: -- Does this need documentation? If so, please add the TODOC-SPARK label. Enable mapjoin hints [Spark Branch] --- Key: HIVE-8911 URL: https://issues.apache.org/jira/browse/HIVE-8911 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Chao Fix For: spark-branch Attachments: HIVE-8911.1-spark.patch, HIVE-8911.2-spark.patch, HIVE-8911.3-spark.patch, HIVE-8911.4-spark.patch, HIVE-8911.5-spark.patch, HIVE-8911.6-spark.patch Currently the big table selection in a mapjoin is based on stats. We should also enable the big-table selection based on hints. See class MapJoinProcessor. This is a logical-optimizer class, so we should be able to re-use this without too many changes to hook up with SparkMapJoinResolver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9079) Hive hangs while failed to get executorCount[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9079: -- Attachment: HIVE-9079.1-spark.patch Hive hangs while failed to get executorCount[Spark Branch] -- Key: HIVE-9079 URL: https://issues.apache.org/jira/browse/HIVE-9079 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Chengxiang Li Labels: Spark-M4 Fix For: spark-branch Attachments: HIVE-9079.1-spark.patch, HIVE-9079.1-spark.patch Hive on Spark get executorCount from RSC to dynamically set reduce number, it use future.get() to wait result, which may hangs forever if remote side failed with no notification. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9096) GenericUDF may be left unclosed in PartitionPrune#visitCall()
Ted Yu created HIVE-9096:
Summary: GenericUDF may be left unclosed in
PartitionPrune#visitCall()
Key: HIVE-9096
URL: https://issues.apache.org/jira/browse/HIVE-9096
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
In
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/PartitionPrune.java
:
{code}
GenericUDF hiveUDF = SqlFunctionConverter.getHiveUDF(call.getOperator(),
call.getType(), call.operands.size());
if (hiveUDF != null
!FunctionRegistry.isDeterministic(hiveUDF)) {
return null;
}
{code}
hiveUDF should be closed upon return.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8911) Enable mapjoin hints [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245672#comment-14245672 ] Chao commented on HIVE-8911: Yes, I think this need doc. Thanks [~leftylev] for reminding me! Enable mapjoin hints [Spark Branch] --- Key: HIVE-8911 URL: https://issues.apache.org/jira/browse/HIVE-8911 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Chao Labels: TODOC-SPARK Fix For: spark-branch Attachments: HIVE-8911.1-spark.patch, HIVE-8911.2-spark.patch, HIVE-8911.3-spark.patch, HIVE-8911.4-spark.patch, HIVE-8911.5-spark.patch, HIVE-8911.6-spark.patch Currently the big table selection in a mapjoin is based on stats. We should also enable the big-table selection based on hints. See class MapJoinProcessor. This is a logical-optimizer class, so we should be able to re-use this without too many changes to hook up with SparkMapJoinResolver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8911) Enable mapjoin hints [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao updated HIVE-8911: --- Labels: TODOC-SPARK (was: ) Enable mapjoin hints [Spark Branch] --- Key: HIVE-8911 URL: https://issues.apache.org/jira/browse/HIVE-8911 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Chao Labels: TODOC-SPARK Fix For: spark-branch Attachments: HIVE-8911.1-spark.patch, HIVE-8911.2-spark.patch, HIVE-8911.3-spark.patch, HIVE-8911.4-spark.patch, HIVE-8911.5-spark.patch, HIVE-8911.6-spark.patch Currently the big table selection in a mapjoin is based on stats. We should also enable the big-table selection based on hints. See class MapJoinProcessor. This is a logical-optimizer class, so we should be able to re-use this without too many changes to hook up with SparkMapJoinResolver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9053) select constant in union all followed by group by gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9053: -- Attachment: HIVE-9053.04.patch select constant in union all followed by group by gives wrong result Key: HIVE-9053 URL: https://issues.apache.org/jira/browse/HIVE-9053 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9053.01.patch, HIVE-9053.02.patch, HIVE-9053.03.patch, HIVE-9053.04.patch Here is the the way to reproduce with q test: select key from (select '1' as key from src union all select key from src)tab group by key; will give OK NULL 1 This is not correct as src contains many other keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9053) select constant in union all followed by group by gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9053: -- Attachment: HIVE-9053.04.patch add a new test case which will still fail on previous patch. select constant in union all followed by group by gives wrong result Key: HIVE-9053 URL: https://issues.apache.org/jira/browse/HIVE-9053 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9053.01.patch, HIVE-9053.02.patch, HIVE-9053.03.patch, HIVE-9053.04.patch Here is the the way to reproduce with q test: select key from (select '1' as key from src union all select key from src)tab group by key; will give OK NULL 1 This is not correct as src contains many other keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9053) select constant in union all followed by group by gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9053: -- Status: Open (was: Patch Available) select constant in union all followed by group by gives wrong result Key: HIVE-9053 URL: https://issues.apache.org/jira/browse/HIVE-9053 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9053.01.patch, HIVE-9053.02.patch, HIVE-9053.03.patch, HIVE-9053.04.patch Here is the the way to reproduce with q test: select key from (select '1' as key from src union all select key from src)tab group by key; will give OK NULL 1 This is not correct as src contains many other keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9053) select constant in union all followed by group by gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9053: -- Attachment: (was: HIVE-9053.04.patch) select constant in union all followed by group by gives wrong result Key: HIVE-9053 URL: https://issues.apache.org/jira/browse/HIVE-9053 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9053.01.patch, HIVE-9053.02.patch, HIVE-9053.03.patch, HIVE-9053.04.patch Here is the the way to reproduce with q test: select key from (select '1' as key from src union all select key from src)tab group by key; will give OK NULL 1 This is not correct as src contains many other keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9053) select constant in union all followed by group by gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9053: -- Status: Patch Available (was: Open) select constant in union all followed by group by gives wrong result Key: HIVE-9053 URL: https://issues.apache.org/jira/browse/HIVE-9053 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9053.01.patch, HIVE-9053.02.patch, HIVE-9053.03.patch, HIVE-9053.04.patch Here is the the way to reproduce with q test: select key from (select '1' as key from src union all select key from src)tab group by key; will give OK NULL 1 This is not correct as src contains many other keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28973: select constant in union all followed by group by gives wrong result
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28973/ --- (Updated Dec. 13, 2014, 11:52 p.m.) Review request for hive, Ashutosh Chauhan and John Pullokkaran. Changes --- add one more test case Repository: hive-git Description --- Here is the the way to reproduce with q test: select key from (select '1' as key from src union all select key from src)tab group by key; will give OK NULL 1 This is not correct as src contains many other keys. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java 8551a5b ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java 957c327 ql/src/test/queries/clientpositive/metadataOnlyOptimizer.q PRE-CREATION ql/src/test/results/clientpositive/metadataOnlyOptimizer.q.out PRE-CREATION ql/src/test/results/clientpositive/optimize_nullscan.q.out 02636ba Diff: https://reviews.apache.org/r/28973/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Commented] (HIVE-9053) select constant in union all followed by group by gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245772#comment-14245772 ] Pengcheng Xiong commented on HIVE-9053: --- [~ashutoshc], I think the test failure is unrelated and the patch is safe to check in. Could you please take a look? Thanks! select constant in union all followed by group by gives wrong result Key: HIVE-9053 URL: https://issues.apache.org/jira/browse/HIVE-9053 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9053.01.patch, HIVE-9053.02.patch, HIVE-9053.03.patch, HIVE-9053.04.patch Here is the the way to reproduce with q test: select key from (select '1' as key from src union all select key from src)tab group by key; will give OK NULL 1 This is not correct as src contains many other keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9055) Tez: union all followed by group by followed by another union all gives error
[ https://issues.apache.org/jira/browse/HIVE-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245773#comment-14245773 ] Pengcheng Xiong commented on HIVE-9055: --- [~jpullokkaran], could you please help check in the patch so that I can test union distinct on Tez? This one blocks any union distinct to run on Tez. Thanks! Tez: union all followed by group by followed by another union all gives error - Key: HIVE-9055 URL: https://issues.apache.org/jira/browse/HIVE-9055 Project: Hive Issue Type: Bug Reporter: Pengcheng Xiong Assignee: Vikram Dixit K Attachments: HIVE-9055.1.patch, HIVE-9055.WIP.patch Here is the way to produce it: in Hive q test setting (with src table) set hive.execution.engine=tez; select key from ( select key from src union all select key from src ) tab group by key union all select key from src; will give you ERROR 2014-12-09 11:38:48,316 ERROR ql.Driver (SessionState.java:printError(834)) - FAILED: IndexOutOfBoundsException Index: -1, Size: 1 java.lang.IndexOutOfBoundsException: Index: -1, Size: 1 at java.util.LinkedList.checkElementIndex(LinkedList.java:553) at java.util.LinkedList.get(LinkedList.java:474) at org.apache.hadoop.hive.ql.parse.GenTezWork.process(GenTezWork.java:354) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:87) at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103) at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103) at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103) at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103) at org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.startWalking(GenTezWorkWalker.java:69) at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:368) at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:202) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10202) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:420) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:306) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1108) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1035) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:199) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:362) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:297) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:834) at org.apache.hadoop.hive.cli.TestMiniTezCliDriver.runTest(TestMiniTezCliDriver.java:136) at org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_uniontez(TestMiniTezCliDriver.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) btw: there is not problem when it is run with MR -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7816) Enable map-join tests which Tez executes [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-7816:
--
Attachment: HIVE-7816.1-spark.patch
Enable map-join tests which Tez executes [Spark Branch]
---
Key: HIVE-7816
URL: https://issues.apache.org/jira/browse/HIVE-7816
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-7816.1-spark.patch, HIVE-7816.1-spark.patch
{noformat}
auto_join0.q,\
auto_join1.q,\
cross_join.q,\
cross_product_check_1.q,\
cross_product_check_2.q,\
{noformat}
{noformat}
filter_join_breaktask.q,\
filter_join_breaktask2.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7816) Enable map-join tests which Tez executes [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-7816:
--
Attachment: (was: HIVE-7816.1-spark.patch)
Enable map-join tests which Tez executes [Spark Branch]
---
Key: HIVE-7816
URL: https://issues.apache.org/jira/browse/HIVE-7816
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-7816.1-spark.patch, HIVE-7816.1-spark.patch
{noformat}
auto_join0.q,\
auto_join1.q,\
cross_join.q,\
cross_product_check_1.q,\
cross_product_check_2.q,\
{noformat}
{noformat}
filter_join_breaktask.q,\
filter_join_breaktask2.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7816) Enable map-join tests which Tez executes [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-7816:
--
Attachment: (was: HIVE-7816.1-spark.patch)
Enable map-join tests which Tez executes [Spark Branch]
---
Key: HIVE-7816
URL: https://issues.apache.org/jira/browse/HIVE-7816
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-7816.1-spark.patch
{noformat}
auto_join0.q,\
auto_join1.q,\
cross_join.q,\
cross_product_check_1.q,\
cross_product_check_2.q,\
{noformat}
{noformat}
filter_join_breaktask.q,\
filter_join_breaktask2.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9078) Hive should not submit second SparkTask while previous one has failed.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9078:
--
Attachment: HIVE-9078.2-spark.patch
Hive should not submit second SparkTask while previous one has failed.[Spark
Branch]
Key: HIVE-9078
URL: https://issues.apache.org/jira/browse/HIVE-9078
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Labels: Spark-M4
Attachments: HIVE-9078.1-spark.patch, HIVE-9078.2-spark.patch
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211135050_51e5ae15-49a3-4a46-826f-e27ee314ccb2
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
Launching Job 2 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
OK
Time taken: 68.53 seconds
{noformat}
2 issue in the above CLI output.
# For a query which would be translated into multi SparkTask, is previous
SparkTask failed, Hive should failed right away, the following SparkTask
should not be submitted any more.
# Print failed info in Hive console while query failed.
The correct CLI output while query failed:
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211142929_ddb7f205-8422-44b4-96bd-96a1c9291895
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9078) Hive should not submit second SparkTask while previous one has failed.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9078:
--
Attachment: (was: HIVE-9078.2-spark.patch)
Hive should not submit second SparkTask while previous one has failed.[Spark
Branch]
Key: HIVE-9078
URL: https://issues.apache.org/jira/browse/HIVE-9078
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Labels: Spark-M4
Attachments: HIVE-9078.1-spark.patch, HIVE-9078.2-spark.patch
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211135050_51e5ae15-49a3-4a46-826f-e27ee314ccb2
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
Launching Job 2 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
OK
Time taken: 68.53 seconds
{noformat}
2 issue in the above CLI output.
# For a query which would be translated into multi SparkTask, is previous
SparkTask failed, Hive should failed right away, the following SparkTask
should not be submitted any more.
# Print failed info in Hive console while query failed.
The correct CLI output while query failed:
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211142929_ddb7f205-8422-44b4-96bd-96a1c9291895
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9078) Hive should not submit second SparkTask while previous one has failed.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9078:
--
Attachment: (was: HIVE-9078.2-spark.patch)
Hive should not submit second SparkTask while previous one has failed.[Spark
Branch]
Key: HIVE-9078
URL: https://issues.apache.org/jira/browse/HIVE-9078
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Labels: Spark-M4
Attachments: HIVE-9078.1-spark.patch, HIVE-9078.2-spark.patch
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211135050_51e5ae15-49a3-4a46-826f-e27ee314ccb2
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
Launching Job 2 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
OK
Time taken: 68.53 seconds
{noformat}
2 issue in the above CLI output.
# For a query which would be translated into multi SparkTask, is previous
SparkTask failed, Hive should failed right away, the following SparkTask
should not be submitted any more.
# Print failed info in Hive console while query failed.
The correct CLI output while query failed:
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211142929_ddb7f205-8422-44b4-96bd-96a1c9291895
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9078) Hive should not submit second SparkTask while previous one has failed.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9078:
--
Attachment: (was: HIVE-9078.1-spark.patch)
Hive should not submit second SparkTask while previous one has failed.[Spark
Branch]
Key: HIVE-9078
URL: https://issues.apache.org/jira/browse/HIVE-9078
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Labels: Spark-M4
Attachments: HIVE-9078.1-spark.patch, HIVE-9078.2-spark.patch
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211135050_51e5ae15-49a3-4a46-826f-e27ee314ccb2
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
Launching Job 2 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
OK
Time taken: 68.53 seconds
{noformat}
2 issue in the above CLI output.
# For a query which would be translated into multi SparkTask, is previous
SparkTask failed, Hive should failed right away, the following SparkTask
should not be submitted any more.
# Print failed info in Hive console while query failed.
The correct CLI output while query failed:
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211142929_ddb7f205-8422-44b4-96bd-96a1c9291895
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9090) Rename Tez File Merge Work to smaller name
[ https://issues.apache.org/jira/browse/HIVE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-9090: Resolution: Fixed Fix Version/s: 0.14.1 0.15.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-0.14.1 Rename Tez File Merge Work to smaller name Key: HIVE-9090 URL: https://issues.apache.org/jira/browse/HIVE-9090 Project: Hive Issue Type: Bug Affects Versions: 0.14.0, 0.15.0, 0.14.1 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Priority: Trivial Labels: tez-ui Fix For: 0.15.0, 0.14.1 Attachments: HIVE-9090.1.patch This is just a cosmetic change. The Tez File Merge Work vertex name is long that in-place update UI showing vertex name is out of place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9078) Hive should not submit second SparkTask while previous one has failed.[Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245837#comment-14245837
]
Hive QA commented on HIVE-9078:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12687095/HIVE-9078.2-spark.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 7233 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_spark4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_hook
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/539/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/539/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-539/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12687095 - PreCommit-HIVE-SPARK-Build
Hive should not submit second SparkTask while previous one has failed.[Spark
Branch]
Key: HIVE-9078
URL: https://issues.apache.org/jira/browse/HIVE-9078
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Chengxiang Li
Labels: Spark-M4
Attachments: HIVE-9078.1-spark.patch, HIVE-9078.2-spark.patch
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211135050_51e5ae15-49a3-4a46-826f-e27ee314ccb2
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
Launching Job 2 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
OK
Time taken: 68.53 seconds
{noformat}
2 issue in the above CLI output.
# For a query which would be translated into multi SparkTask, is previous
SparkTask failed, Hive should failed right away, the following SparkTask
should not be submitted any more.
# Print failed info in Hive console while query failed.
The correct CLI output while query failed:
{noformat}
hive select n_name, c_name from nation, customer where nation.n_nationkey =
customer.c_nationkey limit 10;
Query ID = root_20141211142929_ddb7f205-8422-44b4-96bd-96a1c9291895
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapreduce.job.reduces=number
Status: Failed
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7816) Enable map-join tests which Tez executes [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245858#comment-14245858
]
Hive QA commented on HIVE-7816:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12687094/HIVE-7816.1-spark.patch
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 7237 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/540/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-540/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12687094 - PreCommit-HIVE-SPARK-Build
Enable map-join tests which Tez executes [Spark Branch]
---
Key: HIVE-7816
URL: https://issues.apache.org/jira/browse/HIVE-7816
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-7816.1-spark.patch
{noformat}
auto_join0.q,\
auto_join1.q,\
cross_join.q,\
cross_product_check_1.q,\
cross_product_check_2.q,\
{noformat}
{noformat}
filter_join_breaktask.q,\
filter_join_breaktask2.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7816) Enable map-join tests which Tez executes [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao updated HIVE-7816:
---
Attachment: HIVE-7816.2-spark.patch
Regenerated golden files.
Enable map-join tests which Tez executes [Spark Branch]
---
Key: HIVE-7816
URL: https://issues.apache.org/jira/browse/HIVE-7816
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-7816.1-spark.patch, HIVE-7816.2-spark.patch
{noformat}
auto_join0.q,\
auto_join1.q,\
cross_join.q,\
cross_product_check_1.q,\
cross_product_check_2.q,\
{noformat}
{noformat}
filter_join_breaktask.q,\
filter_join_breaktask2.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7816) Enable map-join tests which Tez executes [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao updated HIVE-7816:
---
Attachment: HIVE-7816.2-spark.patch
Enable map-join tests which Tez executes [Spark Branch]
---
Key: HIVE-7816
URL: https://issues.apache.org/jira/browse/HIVE-7816
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-7816.1-spark.patch, HIVE-7816.2-spark.patch
{noformat}
auto_join0.q,\
auto_join1.q,\
cross_join.q,\
cross_product_check_1.q,\
cross_product_check_2.q,\
{noformat}
{noformat}
filter_join_breaktask.q,\
filter_join_breaktask2.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7816) Enable map-join tests which Tez executes [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao updated HIVE-7816:
---
Attachment: (was: HIVE-7816.2-spark.patch)
Enable map-join tests which Tez executes [Spark Branch]
---
Key: HIVE-7816
URL: https://issues.apache.org/jira/browse/HIVE-7816
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-7816.1-spark.patch, HIVE-7816.2-spark.patch
{noformat}
auto_join0.q,\
auto_join1.q,\
cross_join.q,\
cross_product_check_1.q,\
cross_product_check_2.q,\
{noformat}
{noformat}
filter_join_breaktask.q,\
filter_join_breaktask2.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 28933: HIVE-8131:Support timestamp in Avro
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28933/#review65039 --- ql/src/test/results/clientpositive/avro_timestamp.q.out https://reviews.apache.org/r/28933/#comment107977 Looks like we are truncating the timestamp from millis to nanos because Avro spec expects millis. Any idea why the precision for the partition timestamp column is printed in nanos whenever timestamp is in where clause ? Also maybe add another query in the test with millis in the where clause ? serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java https://reviews.apache.org/r/28933/#comment107976 should be timestamp-milllis per the spec... serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java https://reviews.apache.org/r/28933/#comment107975 Shouldn't the logical type be timestamp-millis according to Avro spec ? See: https://issues.apache.org/jira/secure/attachment/12663245/AVRO-739-update-spec.diff serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java https://reviews.apache.org/r/28933/#comment107974 Test for timestamp instead of Test for date - Mohit Sabharwal On Dec. 11, 2014, 5:57 a.m., cheng xu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28933/ --- (Updated Dec. 11, 2014, 5:57 a.m.) Review request for hive. Repository: hive-git Description --- The patch includes: 1.add timestamp support for AvroSerde 2.add related test cases Diffs - data/files/avro_timestamp.txt PRE-CREATION ql/src/test/queries/clientpositive/avro_timestamp.q PRE-CREATION ql/src/test/results/clientpositive/avro_timestamp.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 07c5ecf serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 7639a2b serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java c8eac89 serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java c84b1a0 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java 8cb2dc3 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java cd5a0fa Diff: https://reviews.apache.org/r/28933/diff/ Testing --- Test passed for added cases Thanks, cheng xu
Re: Review Request 28933: HIVE-8131:Support timestamp in Avro
On Dec. 14, 2014, 7:50 a.m., Mohit Sabharwal wrote: ql/src/test/results/clientpositive/avro_timestamp.q.out, line 101 https://reviews.apache.org/r/28933/diff/1/?file=789140#file789140line101 Looks like we are truncating the timestamp from millis to nanos because Avro spec expects millis. Any idea why the precision for the partition timestamp column is printed in nanos whenever timestamp is in where clause ? Also maybe add another query in the test with millis in the where clause ? Typo... I meant truncating the timestamp from nanos to millis - Mohit --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28933/#review65039 --- On Dec. 11, 2014, 5:57 a.m., cheng xu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28933/ --- (Updated Dec. 11, 2014, 5:57 a.m.) Review request for hive. Repository: hive-git Description --- The patch includes: 1.add timestamp support for AvroSerde 2.add related test cases Diffs - data/files/avro_timestamp.txt PRE-CREATION ql/src/test/queries/clientpositive/avro_timestamp.q PRE-CREATION ql/src/test/results/clientpositive/avro_timestamp.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 07c5ecf serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 7639a2b serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java c8eac89 serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java c84b1a0 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java 8cb2dc3 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java cd5a0fa Diff: https://reviews.apache.org/r/28933/diff/ Testing --- Test passed for added cases Thanks, cheng xu
