[jira] [Commented] (HIVE-16230) Enable CBO in presence of hints
[ https://issues.apache.org/jira/browse/HIVE-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932195#comment-15932195 ] Ashutosh Chauhan commented on HIVE-16230: - Tests passed : https://builds.apache.org/job/PreCommit-HIVE-Build/4232/#showFailuresLink > Enable CBO in presence of hints > --- > > Key: HIVE-16230 > URL: https://issues.apache.org/jira/browse/HIVE-16230 > Project: Hive > Issue Type: Improvement > Components: CBO, Logical Optimizer >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-16230.1.patch, HIVE-16230.2.patch, > HIVE-16230.3.patch, HIVE-16230.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932193#comment-15932193
]
Hive QA commented on HIVE-15573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859491/HIVE-15573.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10447 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=23)
[part_inherit_tbl_props_with_star.q,multi_insert_mixed.q,join_cond_pushdown_3.q,cast_on_constant.q,ba_table_udfs.q,ivyDownload.q,null_column.q,quote1.q,empty_dir_in_table.q,vectorization_part_project.q,combine2_hadoop20.q,infer_join_preds.q,show_conf.q,authorization_create_table_owner_privs.q,bucketsortoptimize_insert_4.q,join_grp_diff_keys.q,insert1.q,custom_input_output_format.q,vectorized_bucketmapjoin1.q,skewjoinopt5.q,groupby_sort_2.q,subquery_alias.q,vector_count_distinct.q,input_part9.q,drop_partitions_filter.q,char_varchar_udf.q,union33.q,authorization_grant_option_role.q,union_lateralview.q,nullgroup4.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct]
(batchId=107)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4242/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4242/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4242/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12859491 - PreCommit-HIVE-Build
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> HIVE-15573.06.patch, screenshot-1.png
>
>
> The non-uniform shuffle disabled murmur hash for the shuffle, due to the
> bucketing requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-15573:
Attachment: HIVE-15573.06.patch
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> HIVE-15573.06.patch, screenshot-1.png
>
>
> The non-uniform shuffle disabled murmur hash for the shuffle, due to the
> bucketing requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-15573:
Status: Patch Available (was: In Progress)
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> HIVE-15573.06.patch, screenshot-1.png
>
>
> The non-uniform shuffle disabled murmur hash for the shuffle, due to the
> bucketing requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-15573:
Status: In Progress (was: Patch Available)
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> screenshot-1.png
>
>
> The non-uniform shuffle disabled murmur hash for the shuffle, due to the
> bucketing requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15979) Support character_length and octet_length
[
https://issues.apache.org/jira/browse/HIVE-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carter Shanklin updated HIVE-15979:
---
Labels: (was: TODOC2.2)
> Support character_length and octet_length
> -
>
> Key: HIVE-15979
> URL: https://issues.apache.org/jira/browse/HIVE-15979
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Fix For: 2.2.0
>
> Attachments: HIVE-15979.1.patch, HIVE-15979.2.patch,
> HIVE-15979.3.patch, HIVE-15979.4.patch, HIVE-15979.5.patch,
> HIVE-15979.6.patch, HIVE-15979.7.patch
>
>
> SQL defines standard ways to get number of characters and octets. SQL
> reference: section 6.28. Example:
> vagrant=# select character_length('欲速则不达');
> character_length
> --
> 5
> (1 row)
> vagrant=# select octet_length('欲速则不达');
> octet_length
> --
>15
> (1 row)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15979) Support character_length and octet_length
[
https://issues.apache.org/jira/browse/HIVE-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932149#comment-15932149
]
Carter Shanklin commented on HIVE-15979:
Added these, removing label (if I can...)
> Support character_length and octet_length
> -
>
> Key: HIVE-15979
> URL: https://issues.apache.org/jira/browse/HIVE-15979
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15979.1.patch, HIVE-15979.2.patch,
> HIVE-15979.3.patch, HIVE-15979.4.patch, HIVE-15979.5.patch,
> HIVE-15979.6.patch, HIVE-15979.7.patch
>
>
> SQL defines standard ways to get number of characters and octets. SQL
> reference: section 6.28. Example:
> vagrant=# select character_length('欲速则不达');
> character_length
> --
> 5
> (1 row)
> vagrant=# select octet_length('欲速则不达');
> octet_length
> --
>15
> (1 row)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932106#comment-15932106
]
Hive QA commented on HIVE-15573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859484/HIVE-15573.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 10422 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=20)
[cp_mj_rc.q,rename_column.q,groupby1_limit.q,union5.q,insert_into1.q,drop_udf.q,alter3.q,cast_qualified_types.q,vectorization_4.q,vectorization_2.q,disable_file_format_check.q,decimal_1_1.q,alter_index.q,correlationoptimizer7.q,annotate_stats_table.q,udf3.q,skewjoinopt9.q,drop_function.q,llap_partitioned.q,exim_23_import_part_authsuccess.q,load_dyn_part10.q,skewjoinopt19.q,ba_table_union.q,index_compact.q,vector_date_1.q,join_merge_multi_expressions.q,varchar_udf1.q,orc_dictionary_threshold.q,update_two_cols.q,rcfile_bigdata.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=21)
[authorization_cli_stdconfigauth.q,timestamp_comparison.q,vector_left_outer_join.q,stats10.q,vector_join_part_col_char.q,nonblock_op_deduplicate.q,udf_testlength2.q,skewjoinopt3.q,udf4.q,partition_multilevels.q,udf_stddev.q,sort_merge_join_desc_2.q,parquet_analyze.q,select_dummy_source.q,alter_varchar2.q,tez_multi_union.q,orc_diff_part_cols.q,default_file_format.q,udf_to_utc_timestamp.q,windowing_ntile.q,autoColumnStats_1.q,alter_rename_partition.q,semijoin.q,authorization_create_func1.q,smb_mapjoin_3.q,timestamp_formats.q,orc_ppd_basic.q,udf_notop.q,input_testxpath3.q,array_map_access_nonconstant.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_non_string_partition]
(batchId=94)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vectorization_div0]
(batchId=95)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vectorization_limit]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct]
(batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_data_types]
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_distinct_2]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_groupby_3]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_orderby_5]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_string_concat]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_17]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_div0]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_shufflejoin]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_timestamp_funcs]
(batchId=109)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4241/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4241/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4241/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12859484 - PreCommit-HIVE-Build
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
>
[jira] [Updated] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-15573:
Status: Patch Available (was: In Progress)
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> screenshot-1.png
>
>
> The non-uniform shuffle disabled murmur hash for the shuffle, due to the
> bucketing requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-15573:
Status: In Progress (was: Patch Available)
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> screenshot-1.png
>
>
> The non-uniform shuffle disabled murmur hash for the shuffle, due to the
> bucketing requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-15573:
Description:
The non-uniform shuffle disabled murmur hash for the shuffle, due to the
bucketing requirements demanding the writable hashcode for the shuffles.
{code}
boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
if (!useUniformHash) {
return false;
}
{code}
This check protects the fast ReduceSink ops from being used in ACID inserts.
A specialized case for the following pattern will make ACID insert much faster.
{code}
Reduce Output Operator
sort order:
Map-reduce partition columns: _col0 (type: bigint)
value expressions:
{code}
!screenshot-1.png!
was:
The ACID shuffle disabled murmur hash for the shuffle, due to the bucketing
requirements demanding the writable hashcode for the shuffles.
{code}
boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
if (!useUniformHash) {
return false;
}
{code}
This check protects the fast ReduceSink ops from being used in ACID inserts.
A specialized case for the following pattern will make ACID insert much faster.
{code}
Reduce Output Operator
sort order:
Map-reduce partition columns: _col0 (type: bigint)
value expressions:
{code}
!screenshot-1.png!
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> screenshot-1.png
>
>
> The non-uniform shuffle disabled murmur hash for the shuffle, due to the
> bucketing requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15573) Vectorization: Non-Uniform shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-15573:
Summary: Vectorization: Non-Uniform shuffle ReduceSink is not specialized
(was: Vectorization: ACID shuffle ReduceSink is not specialized )
> Vectorization: Non-Uniform shuffle ReduceSink is not specialized
> -
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> screenshot-1.png
>
>
> The ACID shuffle disabled murmur hash for the shuffle, due to the bucketing
> requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15573) Vectorization: ACID shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-15573:
Attachment: HIVE-15573.05.patch
> Vectorization: ACID shuffle ReduceSink is not specialized
> --
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: acid-test.svg, HIVE-15573.01.patch, HIVE-15573.02.patch,
> HIVE-15573.03.patch, HIVE-15573.04.patch, HIVE-15573.05.patch,
> screenshot-1.png
>
>
> The ACID shuffle disabled murmur hash for the shuffle, due to the bucketing
> requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12274) Increase width of columns used for general configuration in the metastore.
[
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932024#comment-15932024
]
Naveen Gangam commented on HIVE-12274:
--
Review posted at https://reviews.apache.org/r/57759/. Can you guys please
review? Thanks in advance
> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
> Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.3.patch,
> HIVE-12274.4.patch, HIVE-12274.5.patch, HIVE-12274.example.ddl.hql,
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit
> when processing JSON data that has a large nested schema. The struct
> definition is truncated when inserted into the metastore database column
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined
> configuration values it seems rather limiting to impose such a relatively low
> length bound. One can imagine that valid use cases will arise where
> reasonable parameter/property values exceed the current limit.
> h2. Context
> These limitations were in by the [patch
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
> to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}.
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype.
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good.
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes.
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}}
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated
> virtual columns causes the index key to exceed index key length limits.
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}},
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15126: Resolution: Fixed Status: Resolved (was: Patch Available) Committed. Thanks [~daijy] > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Fix For: 1.2.2 > > Attachments: HIVE-15126.1-branch-1.2.patch, > HIVE-15126.1-branch-1.2.patch, HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16110: Fix Version/s: 2.2.0 > Vectorization: Support 2 Value CASE WHEN instead of fall back to > VectorUDFAdaptor > - > > Key: HIVE-16110 > URL: https://issues.apache.org/jira/browse/HIVE-16110 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch, > HIVE-16110.03.patch, HIVE-16110.04.patch, HIVE-16110.05.patch > > > Vectorize more queries by converting a GenericUDFWhen that has 2 values that > are either a column or a constant into a GenericUDFIf, which has vectorized > classes. This eliminates one case so to speak where we use the > VectorUDFAdaptor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline resolved HIVE-16110. - Resolution: Fixed > Vectorization: Support 2 Value CASE WHEN instead of fall back to > VectorUDFAdaptor > - > > Key: HIVE-16110 > URL: https://issues.apache.org/jira/browse/HIVE-16110 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch, > HIVE-16110.03.patch, HIVE-16110.04.patch, HIVE-16110.05.patch > > > Vectorize more queries by converting a GenericUDFWhen that has 2 values that > are either a column or a constant into a GenericUDFIf, which has vectorized > classes. This eliminates one case so to speak where we use the > VectorUDFAdaptor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931899#comment-15931899 ] Matt McCline commented on HIVE-16110: - Committed to master. > Vectorization: Support 2 Value CASE WHEN instead of fall back to > VectorUDFAdaptor > - > > Key: HIVE-16110 > URL: https://issues.apache.org/jira/browse/HIVE-16110 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch, > HIVE-16110.03.patch, HIVE-16110.04.patch, HIVE-16110.05.patch > > > Vectorize more queries by converting a GenericUDFWhen that has 2 values that > are either a column or a constant into a GenericUDFIf, which has vectorized > classes. This eliminates one case so to speak where we use the > VectorUDFAdaptor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16110: Attachment: HIVE-16110.05.patch Fixup Q file issue -- Hive QA not re-run. > Vectorization: Support 2 Value CASE WHEN instead of fall back to > VectorUDFAdaptor > - > > Key: HIVE-16110 > URL: https://issues.apache.org/jira/browse/HIVE-16110 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch, > HIVE-16110.03.patch, HIVE-16110.04.patch, HIVE-16110.05.patch > > > Vectorize more queries by converting a GenericUDFWhen that has 2 values that > are either a column or a constant into a GenericUDFIf, which has vectorized > classes. This eliminates one case so to speak where we use the > VectorUDFAdaptor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16110: Status: In Progress (was: Patch Available) > Vectorization: Support 2 Value CASE WHEN instead of fall back to > VectorUDFAdaptor > - > > Key: HIVE-16110 > URL: https://issues.apache.org/jira/browse/HIVE-16110 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch, > HIVE-16110.03.patch, HIVE-16110.04.patch > > > Vectorize more queries by converting a GenericUDFWhen that has 2 values that > are either a column or a constant into a GenericUDFIf, which has vectorized > classes. This eliminates one case so to speak where we use the > VectorUDFAdaptor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[
https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931832#comment-15931832
]
Hive QA commented on HIVE-16110:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859469/HIVE-16110.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10475 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id3]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case]
(batchId=120)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4240/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4240/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4240/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12859469 - PreCommit-HIVE-Build
> Vectorization: Support 2 Value CASE WHEN instead of fall back to
> VectorUDFAdaptor
> -
>
> Key: HIVE-16110
> URL: https://issues.apache.org/jira/browse/HIVE-16110
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch,
> HIVE-16110.03.patch, HIVE-16110.04.patch
>
>
> Vectorize more queries by converting a GenericUDFWhen that has 2 values that
> are either a column or a constant into a GenericUDFIf, which has vectorized
> classes. This eliminates one case so to speak where we use the
> VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16110: Status: Patch Available (was: In Progress) > Vectorization: Support 2 Value CASE WHEN instead of fall back to > VectorUDFAdaptor > - > > Key: HIVE-16110 > URL: https://issues.apache.org/jira/browse/HIVE-16110 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch, > HIVE-16110.03.patch, HIVE-16110.04.patch > > > Vectorize more queries by converting a GenericUDFWhen that has 2 values that > are either a column or a constant into a GenericUDFIf, which has vectorized > classes. This eliminates one case so to speak where we use the > VectorUDFAdaptor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16110: Attachment: HIVE-16110.04.patch > Vectorization: Support 2 Value CASE WHEN instead of fall back to > VectorUDFAdaptor > - > > Key: HIVE-16110 > URL: https://issues.apache.org/jira/browse/HIVE-16110 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch, > HIVE-16110.03.patch, HIVE-16110.04.patch > > > Vectorize more queries by converting a GenericUDFWhen that has 2 values that > are either a column or a constant into a GenericUDFIf, which has vectorized > classes. This eliminates one case so to speak where we use the > VectorUDFAdaptor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16110) Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16110: Status: In Progress (was: Patch Available) > Vectorization: Support 2 Value CASE WHEN instead of fall back to > VectorUDFAdaptor > - > > Key: HIVE-16110 > URL: https://issues.apache.org/jira/browse/HIVE-16110 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16110.01.patch, HIVE-16110.02.patch, > HIVE-16110.03.patch, HIVE-16110.04.patch > > > Vectorize more queries by converting a GenericUDFWhen that has 2 values that > are either a column or a constant into a GenericUDFIf, which has vectorized > classes. This eliminates one case so to speak where we use the > VectorUDFAdaptor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15857) Vectorization: Add string conversion case for UDFToInteger, etc
[ https://issues.apache.org/jira/browse/HIVE-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15857: Fix Version/s: 2.2.0 > Vectorization: Add string conversion case for UDFToInteger, etc > --- > > Key: HIVE-15857 > URL: https://issues.apache.org/jira/browse/HIVE-15857 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15857.01.patch, HIVE-15857.02.patch, > HIVE-15857.03.patch, HIVE-15857.04.patch, HIVE-15857.05.patch, > HIVE-15857.06.patch, HIVE-15857.07.patch > > > Otherwise, VectorUDFAdaptor is used to convert a column from String to Int, > etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15857) Vectorization: Add string conversion case for UDFToInteger, etc
[ https://issues.apache.org/jira/browse/HIVE-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15857: Resolution: Fixed Status: Resolved (was: Patch Available) > Vectorization: Add string conversion case for UDFToInteger, etc > --- > > Key: HIVE-15857 > URL: https://issues.apache.org/jira/browse/HIVE-15857 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-15857.01.patch, HIVE-15857.02.patch, > HIVE-15857.03.patch, HIVE-15857.04.patch, HIVE-15857.05.patch, > HIVE-15857.06.patch, HIVE-15857.07.patch > > > Otherwise, VectorUDFAdaptor is used to convert a column from String to Int, > etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15857) Vectorization: Add string conversion case for UDFToInteger, etc
[ https://issues.apache.org/jira/browse/HIVE-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931762#comment-15931762 ] Matt McCline commented on HIVE-15857: - Committed to master. > Vectorization: Add string conversion case for UDFToInteger, etc > --- > > Key: HIVE-15857 > URL: https://issues.apache.org/jira/browse/HIVE-15857 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15857.01.patch, HIVE-15857.02.patch, > HIVE-15857.03.patch, HIVE-15857.04.patch, HIVE-15857.05.patch, > HIVE-15857.06.patch, HIVE-15857.07.patch > > > Otherwise, VectorUDFAdaptor is used to convert a column from String to Int, > etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15857) Vectorization: Add string conversion case for UDFToInteger, etc
[ https://issues.apache.org/jira/browse/HIVE-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15857: Attachment: HIVE-15857.07.patch Fixup a few Q files -- no Hive QA rerun. > Vectorization: Add string conversion case for UDFToInteger, etc > --- > > Key: HIVE-15857 > URL: https://issues.apache.org/jira/browse/HIVE-15857 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15857.01.patch, HIVE-15857.02.patch, > HIVE-15857.03.patch, HIVE-15857.04.patch, HIVE-15857.05.patch, > HIVE-15857.06.patch, HIVE-15857.07.patch > > > Otherwise, VectorUDFAdaptor is used to convert a column from String to Int, > etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14016) Vectorization: Add support for Grouping Sets
[
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931748#comment-15931748
]
Matt McCline commented on HIVE-14016:
-
Committed to master. Thanks [~gopalv] and [~jcamachorodriguez].
> Vectorization: Add support for Grouping Sets
>
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch,
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch,
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch,
> HIVE-14016.092.patch, HIVE-14016.093.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
> if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size();
> groupingSetPos++) {
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
> }
> FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
> // Some keys need to be left to null corresponding to that grouping
> set.
> for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
> }
> newKeysArray[groupingSetsPosition] =
> newKeysGroupingSets[groupingSetPos];
> processKey(row, rowInspector);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12274) Increase width of columns used for general configuration in the metastore.
[
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931690#comment-15931690
]
Naveen Gangam commented on HIVE-12274:
--
Looks much better. The CLOB data types do not support SQL operations like
{{<>}},{{!=}} and {{=}} at least in derby (which is what the test uses). So I
have had to switch it to use {{LIKE}} in the JDO query construction. The tests
that were failing were trying to filter table names based on property values
for TABLE_PARAMS. I have had to disable the 2 tests that did <> and != on the
properties. I will have to get them to use NOT LIKE instead. This requires lot
more work and I have to get the ExpressionTree to be able to parse multi-worded
operators like NOT LIKE and I will file a separate jira to do this.
For now, this looks good to me. All tests are fine. The failures above are due
to missing test output files are appear to be happening to other
builds,immediately preceeding and succeeding this build, as well.
> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
> Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.3.patch,
> HIVE-12274.4.patch, HIVE-12274.5.patch, HIVE-12274.example.ddl.hql,
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit
> when processing JSON data that has a large nested schema. The struct
> definition is truncated when inserted into the metastore database column
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined
> configuration values it seems rather limiting to impose such a relatively low
> length bound. One can imagine that valid use cases will arise where
> reasonable parameter/property values exceed the current limit.
> h2. Context
> These limitations were in by the [patch
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
> to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}.
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype.
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good.
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes.
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}}
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated
> virtual columns causes the index key to exceed index key length limits.
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}},
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation
[jira] [Commented] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[
https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931622#comment-15931622
]
Hive QA commented on HIVE-15126:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859448/HIVE-15126.1-branch-1.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 135 failed/errored test(s), 7884 tests
executed
*Failed tests:*
{noformat}
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=340)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=371)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=350)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=356)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=394)
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=11)
[orc_ppd_decimal.q,input_part1.q,udf_when.q,union14.q,subquery_unqual_corr_expr.q,alter_table_serde.q,drop_view.q,union_remove_4.q,delimiter.q,udaf_collect_set.q,combine3.q,smb_mapjoin_25.q,alter_change_db_location.q,update_all_non_partitioned.q,udf_minute.q]
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=370)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=359)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=379)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=325)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=358)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=328)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=337)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=382)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=332)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=365)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=397)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=398)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=386)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=374)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=373)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=376)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=352)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=342)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=355)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=400)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=401)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=383)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=357)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=429)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=388)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=390)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=426)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=425)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=423)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=428)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=389)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=395)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=396)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=363)
[jira] [Updated] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15126: Attachment: HIVE-15126.1-branch-1.2.patch > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Fix For: 1.2.2 > > Attachments: HIVE-15126.1-branch-1.2.patch, > HIVE-15126.1-branch-1.2.patch, HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15126: Status: Patch Available (was: Reopened) > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Fix For: 1.2.2 > > Attachments: HIVE-15126.1-branch-1.2.patch, HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
[ https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline resolved HIVE-16245. - Resolution: Fixed > Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode > -- > > Key: HIVE-16245 > URL: https://issues.apache.org/jira/browse/HIVE-16245 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16245.01.patch, HIVE-16245.02.patch > > > When the planner is able to make a column a constant, MERGEPARTIAL mode in > VectorGroupByOperator is broken because it doesn't evaluate the key > expression. One result is execution cast exception errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
[ https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16245: Attachment: HIVE-16245.02.patch > Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode > -- > > Key: HIVE-16245 > URL: https://issues.apache.org/jira/browse/HIVE-16245 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16245.01.patch, HIVE-16245.02.patch > > > When the planner is able to make a column a constant, MERGEPARTIAL mode in > VectorGroupByOperator is broken because it doesn't evaluate the key > expression. One result is execution cast exception errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
[ https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931604#comment-15931604 ] Matt McCline commented on HIVE-16245: - Fix UT and not re-run Hive QA. > Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode > -- > > Key: HIVE-16245 > URL: https://issues.apache.org/jira/browse/HIVE-16245 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16245.01.patch, HIVE-16245.02.patch > > > When the planner is able to make a column a constant, MERGEPARTIAL mode in > VectorGroupByOperator is broken because it doesn't evaluate the key > expression. One result is execution cast exception errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16245) Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode
[ https://issues.apache.org/jira/browse/HIVE-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-16245: Status: In Progress (was: Patch Available) > Vectorization: Does not handle non-column key expressions in MERGEPARTIAL mode > -- > > Key: HIVE-16245 > URL: https://issues.apache.org/jira/browse/HIVE-16245 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-16245.01.patch, HIVE-16245.02.patch > > > When the planner is able to make a column a constant, MERGEPARTIAL mode in > VectorGroupByOperator is broken because it doesn't evaluate the key > expression. One result is execution cast exception errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HIVE-14016) Vectorization: Add support for Grouping Sets
[
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline resolved HIVE-14016.
-
Resolution: Fixed
> Vectorization: Add support for Grouping Sets
>
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch,
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch,
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch,
> HIVE-14016.092.patch, HIVE-14016.093.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
> if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size();
> groupingSetPos++) {
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
> }
> FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
> // Some keys need to be left to null corresponding to that grouping
> set.
> for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
> }
> newKeysArray[groupingSetsPosition] =
> newKeysGroupingSets[groupingSetPos];
> processKey(row, rowInspector);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-14016) Vectorization: Add support for Grouping Sets
[
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14016:
Fix Version/s: 2.2.0
> Vectorization: Add support for Grouping Sets
>
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch,
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch,
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch,
> HIVE-14016.092.patch, HIVE-14016.093.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
> if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size();
> groupingSetPos++) {
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
> }
> FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
> // Some keys need to be left to null corresponding to that grouping
> set.
> for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
> }
> newKeysArray[groupingSetsPosition] =
> newKeysGroupingSets[groupingSetPos];
> processKey(row, rowInspector);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-14016) Vectorization: Add support for Grouping Sets
[
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14016:
Status: In Progress (was: Patch Available)
> Vectorization: Add support for Grouping Sets
>
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch,
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch,
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch,
> HIVE-14016.092.patch, HIVE-14016.093.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
> if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size();
> groupingSetPos++) {
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
> }
> FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
> // Some keys need to be left to null corresponding to that grouping
> set.
> for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
> }
> newKeysArray[groupingSetsPosition] =
> newKeysGroupingSets[groupingSetPos];
> processKey(row, rowInspector);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-14016) Vectorization: Add support for Grouping Sets
[
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931598#comment-15931598
]
Matt McCline edited comment on HIVE-14016 at 3/19/17 6:49 AM:
--
Fixup a few Q files as #093 -- and don't re-run Hive QA.
was (Author: mmccline):
Fixup a few Q files as #093
> Vectorization: Add support for Grouping Sets
>
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch,
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch,
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch,
> HIVE-14016.092.patch, HIVE-14016.093.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
> if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size();
> groupingSetPos++) {
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
> }
> FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
> // Some keys need to be left to null corresponding to that grouping
> set.
> for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
> }
> newKeysArray[groupingSetsPosition] =
> newKeysGroupingSets[groupingSetPos];
> processKey(row, rowInspector);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14016) Vectorization: Add support for Grouping Sets
[
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931598#comment-15931598
]
Matt McCline commented on HIVE-14016:
-
Fixup a few Q files as #093
> Vectorization: Add support for Grouping Sets
>
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch,
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch,
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch,
> HIVE-14016.092.patch, HIVE-14016.093.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
> if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size();
> groupingSetPos++) {
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
> }
> FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
> // Some keys need to be left to null corresponding to that grouping
> set.
> for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
> }
> newKeysArray[groupingSetsPosition] =
> newKeysGroupingSets[groupingSetPos];
> processKey(row, rowInspector);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-14016) Vectorization: Add support for Grouping Sets
[
https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14016:
Attachment: HIVE-14016.093.patch
> Vectorization: Add support for Grouping Sets
>
>
> Key: HIVE-14016
> URL: https://issues.apache.org/jira/browse/HIVE-14016
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-14016.01.patch, HIVE-14016.02.patch,
> HIVE-14016.03.patch, HIVE-14016.04.patch, HIVE-14016.05.patch,
> HIVE-14016.06.patch, HIVE-14016.07.patch, HIVE-14016.091.patch,
> HIVE-14016.092.patch, HIVE-14016.093.patch, HIVE-14016.09.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of
> grouping-sets inside vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by
> converting a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
> if (groupingSetsPresent) {
> Object[] newKeysArray = newKeys.getKeyArray();
> Object[] cloneNewKeysArray = new Object[newKeysArray.length];
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
> }
> for (int groupingSetPos = 0; groupingSetPos < groupingSets.size();
> groupingSetPos++) {
> for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
> newKeysArray[keyPos] = null;
> }
> FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
> // Some keys need to be left to null corresponding to that grouping
> set.
> for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
> keyPos = bitset.nextSetBit(keyPos+1)) {
> newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
> }
> newKeysArray[groupingSetsPosition] =
> newKeysGroupingSets[groupingSetPos];
> processKey(row, rowInspector);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15126: Attachment: HIVE-15126.1-branch-1.2.patch > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Fix For: 1.2.2 > > Attachments: HIVE-15126.1-branch-1.2.patch, HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15126: Attachment: (was: HIVE-15082.1-branch-1.2.patch) > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Fix For: 1.2.2 > > Attachments: HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reopened HIVE-15126: - > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Fix For: 1.2.2 > > Attachments: HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15126: Resolution: Fixed Status: Resolved (was: Patch Available) Committed. Thanks [~daijy] > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Fix For: 1.2.2 > > Attachments: HIVE-15082.1-branch-1.2.patch, HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15126) Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q
[ https://issues.apache.org/jira/browse/HIVE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15126: Fix Version/s: 1.2.2 > Branch-1.2: Fix TestCliDriver.join_merge_multi_expressions.q > > > Key: HIVE-15126 > URL: https://issues.apache.org/jira/browse/HIVE-15126 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Blocker > Fix For: 1.2.2 > > Attachments: HIVE-15082.1-branch-1.2.patch, HIVE-15126.1.patch > > > The .out file for join_merge_multi_expressions.q needs to be updated. Current > one does not show Select operator in the plan that explain returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15979) Support character_length and octet_length
[
https://issues.apache.org/jira/browse/HIVE-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931589#comment-15931589
]
Lefty Leverenz commented on HIVE-15979:
---
Doc note: These new functions should be documented, with version information,
in the UDF wikidoc.
* [Hive Operators and UDFs -- String Functions |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions]
Added a TODOC2.2 label.
> Support character_length and octet_length
> -
>
> Key: HIVE-15979
> URL: https://issues.apache.org/jira/browse/HIVE-15979
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15979.1.patch, HIVE-15979.2.patch,
> HIVE-15979.3.patch, HIVE-15979.4.patch, HIVE-15979.5.patch,
> HIVE-15979.6.patch, HIVE-15979.7.patch
>
>
> SQL defines standard ways to get number of characters and octets. SQL
> reference: section 6.28. Example:
> vagrant=# select character_length('欲速则不达');
> character_length
> --
> 5
> (1 row)
> vagrant=# select octet_length('欲速则不达');
> octet_length
> --
>15
> (1 row)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15979) Support character_length and octet_length
[
https://issues.apache.org/jira/browse/HIVE-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-15979:
--
Labels: TODOC2.2 (was: )
> Support character_length and octet_length
> -
>
> Key: HIVE-15979
> URL: https://issues.apache.org/jira/browse/HIVE-15979
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15979.1.patch, HIVE-15979.2.patch,
> HIVE-15979.3.patch, HIVE-15979.4.patch, HIVE-15979.5.patch,
> HIVE-15979.6.patch, HIVE-15979.7.patch
>
>
> SQL defines standard ways to get number of characters and octets. SQL
> reference: section 6.28. Example:
> vagrant=# select character_length('欲速则不达');
> character_length
> --
> 5
> (1 row)
> vagrant=# select octet_length('欲速则不达');
> octet_length
> --
>15
> (1 row)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12274) Increase width of columns used for general configuration in the metastore.
[
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931588#comment-15931588
]
Hive QA commented on HIVE-12274:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859441/HIVE-12274.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10358 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=11)
[skewjoinopt15.q,udf_elt.q,join44.q,index_auto_partitioned.q,inputddl2.q,drop_table.q,udf_testlength.q,auto_join33.q,parquet_ppd_varchar.q,udf_sha2.q,groupby5_map_skew.q,merge4.q,storage_format_descriptor.q,masking_1_newdb.q,vector_leftsemi_mapjoin.q,multi_column_in_single.q,cbo_rp_semijoin.q,char_2.q,repl_1_drop.q,non_ascii_literal2.q,ppd_clusterby.q,load_part_authsuccess.q,auto_sortmerge_join_15.q,explain_rearrange.q,varchar_union1.q,vector_udf2.q,groupby_constcolval.q,annotate_stats_limit.q,explainanalyze_5.q,distinct_windowing.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=19)
[cp_mj_rc.q,order.q,udf_bitwise_shiftleft.q,insert_values_non_partitioned.q,skewjoinopt10.q,extrapolate_part_stats_date.q,udf_sin.q,vectorized_math_funcs.q,join45.q,join14.q,query_result_fileformat.q,orc_merge1.q,cbo_union_view.q,tez_union_decimal.q,nonmr_fetch.q,order_null.q,cbo_rp_views.q,lvj_mapjoin.q,insert_acid_dynamic_partition.q,skewjoinopt6.q,cbo_rp_cross_product_check_2.q,union_remove_19.q,mapreduce7.q,spark_use_op_stats.q,update_two_cols.q,update_where_no_match.q,correlationoptimizer11.q,stats_empty_partition.q,outer_join_ppr.q,authorization_2.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=22)
[multi_insert_mixed.q,bucketcontext_5.q,specialChar.q,union_remove_10.q,skewjoinopt21.q,bucketmapjoin_negative.q,input_part0.q,vector_join_nulls.q,udf_hex.q,reducesink_dedup.q,masking_acid_no_masking.q,smb_mapjoin_4.q,mapreduce2.q,input16.q,udf_in_file.q,vector_empty_where.q,udf_variance.q,join42.q,limit_join_transpose.q,auto_join12.q,skewjoin.q,cte_mat_3.q,parenthesis_star_by.q,vector_decimal_round_2.q,udf_conv.q,column_names_with_leading_and_trailing_spaces.q,vectorized_mapjoin2.q,union_stats.q,nullgroup4.q,authorization_view_disable_cbo_2.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=6)
[ptf_general_queries.q,correlationoptimizer9.q,cross_join_merge.q,parquet_timestamp_conversion.q,sample2.q,parquet_decimal.q,join1.q,bucket_if_with_path_filter.q,join32_lessSize.q,combine2.q,escape3.q,windowing_range_multiorder.q,cte_mat_4.q,udf_weekofyear.q,masking_disablecbo_4.q,char_pad_convert.q,groupby9.q,udaf_covar_samp.q,column_table_stats_orc.q,parquet_columnar.q,skewjoinopt18.q,colstats_all_nulls.q,union_remove_18.q,groupby_duplicate_key.q,pointlookup3.q,orc_remove_cols.q,udf_classloader.q,subq2.q,ctas.q,setop_subq.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=154)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=171)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4238/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4238/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4238/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12859441 - PreCommit-HIVE-Build
> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
> Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.3.patch,
> HIVE-12274.4.patch, HIVE-12274.5.patch, HIVE-12274.example.ddl.hql,
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit
> when processing JSON data that has a large nested schema. The struct
> definition is truncated when inserted into the metastore database column
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in
