Re: Review Request 71932: HIVE-22652
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71932/ --- (Updated Jan. 7, 2020, 7:36 a.m.) Review request for hive and Jesús Camacho Rodríguez. Bugs: HIVE-22652 https://issues.apache.org/jira/browse/HIVE-22652 Repository: hive-git Description --- HIVE-22652: TopNKey push through Group by with Grouping sets Enable TNK op push down through Group by with Grouping sets by removing the lines which checked whether the GBY operator has GROUPING SETS Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java c79c371a8b ql/src/test/queries/clientpositive/topnkey_grouping_sets.q PRE-CREATION ql/src/test/queries/clientpositive/topnkey_grouping_sets_functions.q PRE-CREATION ql/src/test/queries/clientpositive/topnkey_grouping_sets_order.q PRE-CREATION ql/src/test/results/clientpositive/llap/topnkey_grouping_sets.q.out PRE-CREATION ql/src/test/results/clientpositive/llap/topnkey_grouping_sets_functions.q.out PRE-CREATION ql/src/test/results/clientpositive/llap/topnkey_grouping_sets_order.q.out PRE-CREATION ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out c7e837905d ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out d1e8c3806e ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out aa080603e1 ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 59fcf951fe ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 39da7ea903 ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 65d3faa20f ql/src/test/results/clientpositive/perf/tez/constraints/query27.q.out e1a48eaeea ql/src/test/results/clientpositive/perf/tez/constraints/query5.q.out 13288d28b4 ql/src/test/results/clientpositive/perf/tez/constraints/query77.q.out c2758b7033 ql/src/test/results/clientpositive/perf/tez/constraints/query80.q.out 72a54928c2 ql/src/test/results/clientpositive/perf/tez/query14.q.out 00bc4cb026 ql/src/test/results/clientpositive/perf/tez/query27.q.out 774c0fd192 ql/src/test/results/clientpositive/perf/tez/query5.q.out 03980ac2c0 ql/src/test/results/clientpositive/perf/tez/query77.q.out fcfc5a33bc ql/src/test/results/clientpositive/perf/tez/query80.q.out 3020b58781 ql/src/test/results/clientpositive/topnkey_grouping_sets.q.out PRE-CREATION ql/src/test/results/clientpositive/topnkey_grouping_sets_functions.q.out PRE-CREATION ql/src/test/results/clientpositive/topnkey_grouping_sets_order.q.out PRE-CREATION Diff: https://reviews.apache.org/r/71932/diff/6/ Changes: https://reviews.apache.org/r/71932/diff/5-6/ Testing --- - New q test: topnkey_grouping_sets.q - Run `src/test/queries/clientpositive/perf/query*.q` tests with TestTezPerfCliDriver, TestTezPerfConstraintsCliDriver Thanks, Krisztian Kasa
[jira] [Created] (HIVE-22699) Mask UDFs should mask numeric value 0
Quanlong Huang created HIVE-22699:
-
Summary: Mask UDFs should mask numeric value 0
Key: HIVE-22699
URL: https://issues.apache.org/jira/browse/HIVE-22699
Project: Hive
Issue Type: Bug
Reporter: Quanlong Huang
These queries all return 0:
{code:java}
select mask(0);
select mask_first_n(0);
select mask_last_n(0);
select mask_show_first_n(0, 0);
select mask_show_last_n(0, 0);{code}
According to the doc, they should return 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22698) Support Statement#closeOnCompletion()
Iwao AVE! created HIVE-22698: Summary: Support Statement#closeOnCompletion() Key: HIVE-22698 URL: https://issues.apache.org/jira/browse/HIVE-22698 Project: Hive Issue Type: Task Components: JDBC Affects Versions: 2.3.6, 3.1.2 Reporter: Iwao AVE! I am a member of [MyBatis|https://github.com/mybatis] team and a user [reported|https://github.com/mybatis/mybatis-3/issues/1351#issuecomment-523771341] that Hive does not support java.sql.Statement#closeOnCompletion() yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22697) Remove expensive RDBMS queries in lock acquisition logic
Ashutosh Chauhan created HIVE-22697: --- Summary: Remove expensive RDBMS queries in lock acquisition logic Key: HIVE-22697 URL: https://issues.apache.org/jira/browse/HIVE-22697 Project: Hive Issue Type: Improvement Components: Transactions Reporter: Gopal Vijayaraghavan There are queries like "update NEXT_LOCK_ID set nl_next = 163084841;" which can be entirely skipped the ACID impl doesn't require globally unique lock ids because locks are marked as txn.id In a particular instance we observed this logic taking multiple seconds in mysql -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22696) Break up DDLSemanticAnalyzer - extract Table partition analyzers
Miklos Gergely created HIVE-22696: - Summary: Break up DDLSemanticAnalyzer - extract Table partition analyzers Key: HIVE-22696 URL: https://issues.apache.org/jira/browse/HIVE-22696 Project: Hive Issue Type: Sub-task Reporter: Miklos Gergely Assignee: Miklos Gergely DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is to refactor it in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: * have a separate class for each analyzers * have a package for each operation, containing an analyzer, a description, and an operation, so the amount of classes under a package is more manageable Step #11: extract the table column related analyzers from DDLSemanticAnalyzer, and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71761: HIVE-22489
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71761/ --- (Updated Jan. 6, 2020, 2:24 p.m.) Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich. Bugs: HIVE-22489 https://issues.apache.org/jira/browse/HIVE-22489 Repository: hive-git Description --- Reduce Sink operator orders nulls first === 1. Set the default null sort order by hive config when creating Reduce Sink Desc. 2. Hash join uses `org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableSerializeWrite` or `BinarySortableDeserializeRead` for selializing keys. For bigtable keys always ascending and nulls first ordering was hardcoded. This patch changes this behaviour to use the `Operator.getConf().TableDesc.getProperties()` (in this case `MapJoinOperator`) to setup ordering in `BinarySortableSerializeWrite` 3. Use null ordering set in ReduceRecordSource at Reduce phase when comparing keys in `CommonMergeJoinOperator` (This is the null ordering of the children Reduce Sink operators) Diffs (updated) - accumulo-handler/src/test/results/positive/accumulo_queries.q.out 7c552621f2 contrib/src/test/results/clientpositive/udaf_example_group_concat.q.out 6846720d95 hbase-handler/src/test/results/positive/hbase_queries.q.out a32ef81a7b itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out e997fa65cf kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 73fc3e514f ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java 3974627a24 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 72446afeda ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java 2380d936f2 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyMultiKeyOperator.java f587517b08 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerMultiKeyOperator.java cdee3fd957 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiMultiKeyOperator.java e5d9fdae19 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterMultiKeyOperator.java 29c531bd51 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java a4cda921a5 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java 43f093d906 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java 8dce5b82d3 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java a35401d9b2 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringCommon.java 1b108a8c14 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java 446feb2526 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java c28ef9be2b ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java 17bd5fda93 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java 4ab8902a3f ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java 21c355cb42 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongCommon.java de1ee15c3b ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMap.java 42573f0898 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMultiSet.java 829a03737d ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashSet.java 18e1435019 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringCommon.java da0e8365b1 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMap.java 6c4d8a81d1 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMultiSet.java a6b754c7eb ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java fdcd83dde7 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 5c409e4573 ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java a50ad78e8f ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java 0f95d7788c ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 89b55001f0
Re: Review Request 71761: HIVE-22489
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71761/ --- (Updated Jan. 6, 2020, 12:23 p.m.) Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich. Bugs: HIVE-22489 https://issues.apache.org/jira/browse/HIVE-22489 Repository: hive-git Description --- Reduce Sink operator orders nulls first === 1. Set the default null sort order by hive config when creating Reduce Sink Desc. 2. Hash join uses `org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableSerializeWrite` or `BinarySortableDeserializeRead` for selializing keys. For bigtable keys always ascending and nulls first ordering was hardcoded. This patch changes this behaviour to use the `Operator.getConf().TableDesc.getProperties()` (in this case `MapJoinOperator`) to setup ordering in `BinarySortableSerializeWrite` 3. Use null ordering set in ReduceRecordSource at Reduce phase when comparing keys in `CommonMergeJoinOperator` (This is the null ordering of the children Reduce Sink operators) Diffs (updated) - accumulo-handler/src/test/results/positive/accumulo_queries.q.out 7c552621f2 contrib/src/test/results/clientpositive/udaf_example_group_concat.q.out 6846720d95 hbase-handler/src/test/results/positive/hbase_queries.q.out a32ef81a7b itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out e997fa65cf kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 73fc3e514f ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java 3974627a24 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 72446afeda ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java 2380d936f2 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyMultiKeyOperator.java f587517b08 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerMultiKeyOperator.java cdee3fd957 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiMultiKeyOperator.java e5d9fdae19 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterMultiKeyOperator.java 29c531bd51 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java a4cda921a5 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java 43f093d906 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java 8dce5b82d3 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java a35401d9b2 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringCommon.java 1b108a8c14 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java 446feb2526 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java c28ef9be2b ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java 17bd5fda93 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java 4ab8902a3f ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java 21c355cb42 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongCommon.java de1ee15c3b ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMap.java 42573f0898 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMultiSet.java 829a03737d ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashSet.java 18e1435019 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringCommon.java da0e8365b1 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMap.java 6c4d8a81d1 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMultiSet.java a6b754c7eb ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java fdcd83dde7 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java 5c409e4573 ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java a50ad78e8f ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java 0f95d7788c ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 89b55001f0
Re: Review Request 71932: HIVE-22652
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71932/ --- (Updated Jan. 6, 2020, 9:57 a.m.) Review request for hive and Jesús Camacho Rodríguez. Bugs: HIVE-22652 https://issues.apache.org/jira/browse/HIVE-22652 Repository: hive-git Description --- HIVE-22652: TopNKey push through Group by with Grouping sets Enable TNK op push down through Group by with Grouping sets by removing the lines which checked whether the GBY operator has GROUPING SETS Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java c79c371a8b ql/src/test/queries/clientpositive/topnkey_grouping_sets.q PRE-CREATION ql/src/test/results/clientpositive/llap/topnkey_grouping_sets.q.out PRE-CREATION ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out c7e837905d ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out d1e8c3806e ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out aa080603e1 ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 59fcf951fe ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 39da7ea903 ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 65d3faa20f ql/src/test/results/clientpositive/perf/tez/constraints/query27.q.out e1a48eaeea ql/src/test/results/clientpositive/perf/tez/constraints/query5.q.out 13288d28b4 ql/src/test/results/clientpositive/perf/tez/constraints/query77.q.out c2758b7033 ql/src/test/results/clientpositive/perf/tez/constraints/query80.q.out 72a54928c2 ql/src/test/results/clientpositive/perf/tez/query14.q.out 00bc4cb026 ql/src/test/results/clientpositive/perf/tez/query27.q.out 774c0fd192 ql/src/test/results/clientpositive/perf/tez/query5.q.out 03980ac2c0 ql/src/test/results/clientpositive/perf/tez/query77.q.out fcfc5a33bc ql/src/test/results/clientpositive/perf/tez/query80.q.out 3020b58781 ql/src/test/results/clientpositive/topnkey_grouping_sets.q.out PRE-CREATION Diff: https://reviews.apache.org/r/71932/diff/5/ Changes: https://reviews.apache.org/r/71932/diff/4-5/ Testing --- - New q test: topnkey_grouping_sets.q - Run `src/test/queries/clientpositive/perf/query*.q` tests with TestTezPerfCliDriver, TestTezPerfConstraintsCliDriver Thanks, Krisztian Kasa
[jira] [Created] (HIVE-22695) DecimalColumnVector setElement throws class cast exception if input is of type LongColumnVector
mahesh kumar behera created HIVE-22695: -- Summary: DecimalColumnVector setElement throws class cast exception if input is of type LongColumnVector Key: HIVE-22695 URL: https://issues.apache.org/jira/browse/HIVE-22695 Project: Hive Issue Type: Bug Reporter: mahesh kumar behera Assignee: mahesh kumar behera Before casting the input to decimal type, the type should be checked. For long and double type, the value should be extracted and from that decimal type should be created. -- This message was sent by Atlassian Jira (v8.3.4#803005)
