[jira] [Updated] (HIVE-21697) Remove periodical full refresh in HMS cache

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21697:
--
Status: Patch Available  (was: Open)

> Remove periodical full refresh in HMS cache
> ---
>
> Key: HIVE-21697
> URL: https://issues.apache.org/jira/browse/HIVE-21697
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21697.1.patch
>
>
> In HIVE-18661, we added periodical notification based refresh in HMS cache. 
> We shall remove periodical full refresh to simplify the code as it will no 
> longer be used. In the mean time, we introduced mechanism to provide 
> monotonic reads through the CachedStore.commitTransaction. This will no 
> longer be needed after HIVE-21637. So I will remove related code as well. 
> This will provide some performance benefits include:
> 1. We don't have to slow down write to catch up notification logs. Write can 
> be done immediately and tag the cache with writeids
> 2. We can read from cache even if updateUsingNotificationEvents is running. 
> Read will compare the writeids of the cache so monotonic reads will be 
> guaranteed
> I'd like to put a patch separately with HIVE-21637 so it can be tested 
> independently. HMW will use periodical notification based refresh to update 
> cache. And it will temporarily lift the monotonic reads guarantee until 
> HIVE-21637 checkin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21697) Remove periodical full refresh in HMS cache

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21697:
--
Attachment: HIVE-21697.1.patch

> Remove periodical full refresh in HMS cache
> ---
>
> Key: HIVE-21697
> URL: https://issues.apache.org/jira/browse/HIVE-21697
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21697.1.patch
>
>
> In HIVE-18661, we added periodical notification based refresh in HMS cache. 
> We shall remove periodical full refresh to simplify the code as it will no 
> longer be used. In the mean time, we introduced mechanism to provide 
> monotonic reads through the CachedStore.commitTransaction. This will no 
> longer be needed after HIVE-21637. So I will remove related code as well. 
> This will provide some performance benefits include:
> 1. We don't have to slow down write to catch up notification logs. Write can 
> be done immediately and tag the cache with writeids
> 2. We can read from cache even if updateUsingNotificationEvents is running. 
> Read will compare the writeids of the cache so monotonic reads will be 
> guaranteed
> I'd like to put a patch separately with HIVE-21637 so it can be tested 
> independently. HMW will use periodical notification based refresh to update 
> cache. And it will temporarily lift the monotonic reads guarantee until 
> HIVE-21637 checkin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Shubham Chaurasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shubham Chaurasia updated HIVE-21641:
-
Attachment: HIVE-21641.5.branch-3.patch

> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: Branch3Candidate, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch, HIVE-21641.4.patch, HIVE-21641.5.branch-3.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> |   my_avg   |
> ++
> | 37.8923531030581611189434  |
> ++
> {code} 
> Result from Llap external client
> {code}
> +-+
> |   my_avg|
> +-+
> |37.892353|
> +-+
> {code}
>  
> This is due to Driver(beeline path) calls 
> [analyzeInternal()|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L328]
>  for getting result set schema which initializes 
> [resultSchema|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L333]
>  after some more transformations as compared to llap-ext-client which calls 
> [genLogicalPlan()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L561]
> Replacing {{genLogicalPlan()}} by {{analyze()}} resolves this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21694) Hive driver wait time is fixed for task getting executed in parallel.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834400#comment-16834400
 ] 

Hive QA commented on HIVE-21694:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967957/HIVE-21694.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15918 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=28)

[acid_vectorization_missing_cols.q,cte_7.q,cli_print_escape_crlf.q,alter_partition_coltype.q,orc_diff_part_cols2.q,cbo_rp_auto_join17.q,temp_table_truncate.q,float_equality.q,masking_6.q,skewjoinopt4.q,count.q,schema_evol_orc_vec_part_all_primitive_llap_io.q,parallel.q,alter_partition_change_col.q,groupby_grouping_sets2.q,truncate_column_buckets.q,infer_bucket_sort_list_bucket.q,vectorization_10.q,dbtxnmgr_query5.q,interval_udf.q,udaf_ngrams.q,spark_dynamic_partition_pruning.q,partition_condition_remover.q,ppd2.q,hybridgrace_hashjoin_2.q,vector_varchar_mapjoin1.q,llap_nullscan.q,vectorized_insert_into_bucketed_table.q,list_bucket_dml_13.q,vector_bucket.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=37)

[constantPropWhen.q,druid_timestamptz2.q,udf_floor.q,join20.q,authorization_view_3.q,cross_prod_4.q,orc_ppd_schema_evol_1b.q,exec_parallel_column_stats.q,nested_column_pruning.q,merge_dynamic_partition4.q,cte_3.q,extrapolate_part_stats_full.q,udaf_percentile_approx_23.q,nullability_transitive_inference.q,vector_ptf_part_simple.q,orc_merge6.q,vector_non_string_partition.q,udf_case.q,columnStatsUpdateForStatsOptimizer_1.q,parquet_array_map_emptynullvals.q,udf_length.q,orc_ppd_exception.q,sharedworkext.q,groupby_bigdata.q,concatenate_inherit_table_location.q,parallel_colstats.q,timestamp_4.q,udf7.q,innerjoin.q,parquet_vectorization_pushdown.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=124)

[bucketsortoptimize_insert_4.q,skewjoinopt8.q,vectorization_10.q,auto_join18_multi_distinct.q,custom_input_output_format.q,insert_into3.q,vectorization_6.q,skewjoinopt4.q,count.q,stats9.q,auto_join7.q,parallel.q,timestamp_1.q,union33.q,union_lateralview.q]
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17144/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17144/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17144/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967957 - PreCommit-HIVE-Build

> Hive driver wait time is fixed for task getting executed in parallel.
> -
>
> Key: HIVE-21694
> URL: https://issues.apache.org/jira/browse/HIVE-21694
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21694.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> During a command execution hive driver executes the task in a separate thread 
> if the task to be executed is set as parallel. After starting the task, 
> driver checks if the task has finished execution or not. If the task 
> execution is not finished it waits for 2 seconds before waking up again to 
> check the task status. In case of task with execution time in milliseconds, 
> this wait time can induce substantial overhead. So instead of fixed wait 
> time, exponential backedup sleep time can be used to reduce the sleep 
> overhead. The sleep time can start with 100ms and can increase up to 2 
> seconds doubling on each iteration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-21641:
---
Labels: Branch3Candidate pull-request-available  (was: 
pull-request-available)

> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: Branch3Candidate, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch, HIVE-21641.4.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> |   my_avg   |
> ++
> | 37.8923531030581611189434  |
> ++
> {code} 
> Result from Llap external client
> {code}
> +-+
> |   my_avg|
> +-+
> |37.892353|
> +-+
> {code}
>  
> This is due to Driver(beeline path) calls 
> [analyzeInternal()|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L328]
>  for getting result set schema which initializes 
> [resultSchema|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L333]
>  after some more transformations as compared to llap-ext-client which calls 
> [genLogicalPlan()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L561]
> Replacing {{genLogicalPlan()}} by {{analyze()}} resolves this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-21641:
---
Affects Version/s: 3.1.1

> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch, HIVE-21641.4.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> |   my_avg   |
> ++
> | 37.8923531030581611189434  |
> ++
> {code} 
> Result from Llap external client
> {code}
> +-+
> |   my_avg|
> +-+
> |37.892353|
> +-+
> {code}
>  
> This is due to Driver(beeline path) calls 
> [analyzeInternal()|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L328]
>  for getting result set schema which initializes 
> [resultSchema|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L333]
>  after some more transformations as compared to llap-ext-client which calls 
> [genLogicalPlan()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L561]
> Replacing {{genLogicalPlan()}} by {{analyze()}} resolves this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834382#comment-16834382
 ] 

Gopal V commented on HIVE-21641:


LGTM - +1

> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch, HIVE-21641.4.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> |   my_avg   |
> ++
> | 37.8923531030581611189434  |
> ++
> {code} 
> Result from Llap external client
> {code}
> +-+
> |   my_avg|
> +-+
> |37.892353|
> +-+
> {code}
>  
> This is due to Driver(beeline path) calls 
> [analyzeInternal()|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L328]
>  for getting result set schema which initializes 
> [resultSchema|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L333]
>  after some more transformations as compared to llap-ext-client which calls 
> [genLogicalPlan()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L561]
> Replacing {{genLogicalPlan()}} by {{analyze()}} resolves this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Shubham Chaurasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shubham Chaurasia updated HIVE-21641:
-
Attachment: HIVE-21641.4.patch

> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch, HIVE-21641.4.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> |   my_avg   |
> ++
> | 37.8923531030581611189434  |
> ++
> {code} 
> Result from Llap external client
> {code}
> +-+
> |   my_avg|
> +-+
> |37.892353|
> +-+
> {code}
>  
> This is due to Driver(beeline path) calls 
> [analyzeInternal()|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L328]
>  for getting result set schema which initializes 
> [resultSchema|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L333]
>  after some more transformations as compared to llap-ext-client which calls 
> [genLogicalPlan()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L561]
> Replacing {{genLogicalPlan()}} by {{analyze()}} resolves this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21694) Hive driver wait time is fixed for task getting executed in parallel.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834379#comment-16834379
 ] 

Hive QA commented on HIVE-21694:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
0s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17144/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17144/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Hive driver wait time is fixed for task getting executed in parallel.
> -
>
> Key: HIVE-21694
> URL: https://issues.apache.org/jira/browse/HIVE-21694
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21694.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> During a command execution hive driver executes the task in a separate thread 
> if the task to be executed is set as parallel. After starting the task, 
> driver checks if the task has finished execution or not. If the task 
> execution is not finished it waits for 2 seconds before waking up again to 
> check the task status. In case of task with execution time in milliseconds, 
> this wait time can induce substantial overhead. So instead of fixed wait 
> time, exponential backedup sleep time can be used to reduce the sleep 
> overhead. The sleep time can start with 100ms 

[jira] [Commented] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834374#comment-16834374
 ] 

Gopal V commented on HIVE-21641:


{code}
2019-05-06T11:23:08,785 ERROR [Thread-394] server.HiveServer2: Error starting 
Web UI: 
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_102]
at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_102]
at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_102]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock
{code}

flaky test is failing on a port collision.

> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> |   my_avg   |
> ++
> | 37.8923531030581611189434  |
> ++
> {code} 
> Result from Llap external client
> {code}
> +-+
> |   my_avg|
> +-+
> |37.892353|
> +-+
> {code}
>  
> This is due to Driver(beeline path) calls 
> [analyzeInternal()|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L328]
>  for getting result set schema which initializes 
> [resultSchema|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L333]
>  after some more transformations as compared to llap-ext-client which calls 
> [genLogicalPlan()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L561]
> Replacing {{genLogicalPlan()}} by {{analyze()}} resolves this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21625) Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21625:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Patch pushed to master. Thanks Jason for review!

> Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method
> -
>
> Key: HIVE-21625
> URL: https://issues.apache.org/jira/browse/HIVE-21625
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21625.1.patch, HIVE-21625.2.patch, 
> HIVE-21625.3.patch, HIVE-21625.4.patch
>
>
> TxnIdUtils.checkEquivalentWriteIds has a bug which thinks (\{1,2,3,4\}, 6) 
> and (\{1,2,3,4,5,6\}, 8) compatible (the notation is (invalidlist, hwm)). 
> Here is a patch to fix it, also provide a comparison method to check which is 
> newer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-15814) ODBC - LOAD DATA - NullPointerException

[Dan Horne (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834366#comment-16834366
 ] 

Dan Horne edited comment on HIVE-15814 at 5/7/19 4:27 AM:
--

Hi - I hope I'm not hijacking this thread, but I'm encountering a similar issue 
using ODBC and Perl (i.e. some of the same errors in the OP's log), so I 
thought it would be useful to add my "reproduceable" test here.

Strawberry Perl 5.28.1 in Windows Server 2012

Microsoft Hive ODBC Driver 2.01.09.1014

Apache Hive: 2.1.1-mapr-1901-r2

 
{noformat}
use strict;
use DBI;
use autodie;

 my $dbh = DBI->connect(
    'DBI:ODBC:DSN=',
    ‘’,
    '',
    {
    RaiseError   => 1,
        PrintError   => 0,
    }
);

$dbh->do('set role admin;');
{noformat}

 Running it produces the error which were in the OP logs
{quote}DBD::ODBC::db do failed: [Microsoft][Hardy] (61) Server returned error 
with no error message during operation: GetResultSetMetadata

TStatus.statusCode=ERROR_STATUS

    
TStatus.infoMessages="*java.lang.NullPointerException:null:16:15""org.apache.hive.service.cli.TableSchema::TableSchema.java:52""org.apache.hive.service.cli.TableSchema::TableSchema.java:59""org.apache.hive.service.cli.operation.SQLOperation:getResultSetSchema:SQLOperation.java:446""org.apache.hive.service.cli.operation.OperationManager:getOperationR
 (SQL-HY000)
{quote}
Perhaps the error occurs because "set role admin" doesn't return a resultset. 
Issuing regular SQL statements works fine

 

 


was (Author: horned):
Hi - I hope I'm not hijacking this thread, but I'm encountering a similar issue 
using ODBC and Perl (i.e. some of the same errors in the OP's log), so I 
thought it would be useful to add my "reproduceable" test here.

Strawberry Perl 5.28.1 in Windows Server 2012

Microsoft Hive ODBC Driver 2.01.09.1014

Apache Hive: 2.1.1-mapr-1901-r2

 
{quote}use strict;

use DBI;

use autodie;

 

my $dbh = DBI->connect(

    'DBI:ODBC:DSN=',

    ‘’,

    '',

    {

    RaiseError   => 1,

        PrintError   => 0,

    }

);

 

$dbh->do('set role admin;');
{quote}
 Running it produces the error which were in the OP logs
{quote}DBD::ODBC::db do failed: [Microsoft][Hardy] (61) Server returned error 
with no error message during operation: GetResultSetMetadata

TStatus.statusCode=ERROR_STATUS

    
TStatus.infoMessages="*java.lang.NullPointerException:null:16:15""org.apache.hive.service.cli.TableSchema::TableSchema.java:52""org.apache.hive.service.cli.TableSchema::TableSchema.java:59""org.apache.hive.service.cli.operation.SQLOperation:getResultSetSchema:SQLOperation.java:446""org.apache.hive.service.cli.operation.OperationManager:getOperationR
 (SQL-HY000)
{quote}
Perhaps the error occurs because "set role admin" doesn't return a resultset. 
Issuing regular SQL statements works fine

 

 

> ODBC - LOAD DATA - NullPointerException
> ---
>
> Key: HIVE-15814
> URL: https://issues.apache.org/jira/browse/HIVE-15814
> Project: Hive
>  Issue Type: Bug
>  Components: ODBC
>Affects Versions: 1.2.1
>Reporter: Pierre Villard
>Priority: Major
>
> Tested with :
> - Hive 1.2.1.2.4.0
> - Hive 1.2.1.2.5.3
> Information regarding ODBC access:
> - Aginity Workbench for Hadoop 
> - Version 4.8.0.2551 (build 6/14/2016) 
> - Microsoft Windows NT 6.1.7601 Service Pack 1 (64-bit) 
> - DBMS Version: 1.2.1000.2.5.3.0-37 Apache Hive 
> - Driver Used: Hive Native 
> - ODBC Driver Signature: {Aginity Hive}
> Request executed:
> {noformat}
> LOAD DATA INPATH '/tmp/AYX170201a327cc8faf8e562f5eb86225dd1fbf57' INTO TABLE 
> tmp.testpierre
> {noformat}
> Stacktrace available here:
> https://gist.github.com/pvillard31/9a08650f9a80ef0fa7e161d71c0d9e9b
> It seems to be specific to this driver shipped with Aginity studio. Same 
> tests with Simba ODBC driver or HWX ODBC driver are working fine.
> I suppose the NPE could be handled in a proper way even though it seems to be 
> related to some strange calls from ODBC driver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (HIVE-15814) ODBC - LOAD DATA - NullPointerException

[Dan Horne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dan Horne updated HIVE-15814:
-
Comment: was deleted

(was: Please ignore "Unknown macro". Jira is obviously interpreting the baces 
in the code, but there doesn't seem to be a style that avoided this and still 
left the code readable.)

> ODBC - LOAD DATA - NullPointerException
> ---
>
> Key: HIVE-15814
> URL: https://issues.apache.org/jira/browse/HIVE-15814
> Project: Hive
>  Issue Type: Bug
>  Components: ODBC
>Affects Versions: 1.2.1
>Reporter: Pierre Villard
>Priority: Major
>
> Tested with :
> - Hive 1.2.1.2.4.0
> - Hive 1.2.1.2.5.3
> Information regarding ODBC access:
> - Aginity Workbench for Hadoop 
> - Version 4.8.0.2551 (build 6/14/2016) 
> - Microsoft Windows NT 6.1.7601 Service Pack 1 (64-bit) 
> - DBMS Version: 1.2.1000.2.5.3.0-37 Apache Hive 
> - Driver Used: Hive Native 
> - ODBC Driver Signature: {Aginity Hive}
> Request executed:
> {noformat}
> LOAD DATA INPATH '/tmp/AYX170201a327cc8faf8e562f5eb86225dd1fbf57' INTO TABLE 
> tmp.testpierre
> {noformat}
> Stacktrace available here:
> https://gist.github.com/pvillard31/9a08650f9a80ef0fa7e161d71c0d9e9b
> It seems to be specific to this driver shipped with Aginity studio. Same 
> tests with Simba ODBC driver or HWX ODBC driver are working fine.
> I suppose the NPE could be handled in a proper way even though it seems to be 
> related to some strange calls from ODBC driver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-15814) ODBC - LOAD DATA - NullPointerException

[Dan Horne (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834367#comment-16834367
 ] 

Dan Horne commented on HIVE-15814:
--

Please ignore "Unknown macro". Jira is obviously interpreting the baces in the 
code, but there doesn't seem to be a style that avoided this and still left the 
code readable.

> ODBC - LOAD DATA - NullPointerException
> ---
>
> Key: HIVE-15814
> URL: https://issues.apache.org/jira/browse/HIVE-15814
> Project: Hive
>  Issue Type: Bug
>  Components: ODBC
>Affects Versions: 1.2.1
>Reporter: Pierre Villard
>Priority: Major
>
> Tested with :
> - Hive 1.2.1.2.4.0
> - Hive 1.2.1.2.5.3
> Information regarding ODBC access:
> - Aginity Workbench for Hadoop 
> - Version 4.8.0.2551 (build 6/14/2016) 
> - Microsoft Windows NT 6.1.7601 Service Pack 1 (64-bit) 
> - DBMS Version: 1.2.1000.2.5.3.0-37 Apache Hive 
> - Driver Used: Hive Native 
> - ODBC Driver Signature: {Aginity Hive}
> Request executed:
> {noformat}
> LOAD DATA INPATH '/tmp/AYX170201a327cc8faf8e562f5eb86225dd1fbf57' INTO TABLE 
> tmp.testpierre
> {noformat}
> Stacktrace available here:
> https://gist.github.com/pvillard31/9a08650f9a80ef0fa7e161d71c0d9e9b
> It seems to be specific to this driver shipped with Aginity studio. Same 
> tests with Simba ODBC driver or HWX ODBC driver are working fine.
> I suppose the NPE could be handled in a proper way even though it seems to be 
> related to some strange calls from ODBC driver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-15814) ODBC - LOAD DATA - NullPointerException

[Dan Horne (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834366#comment-16834366
 ] 

Dan Horne commented on HIVE-15814:
--

Hi - I hope I'm not hijacking this thread, but I'm encountering a similar issue 
using ODBC and Perl (i.e. some of the same errors in the OP's log), so I 
thought it would be useful to add my "reproduceable" test here.

Strawberry Perl 5.28.1 in Windows Server 2012

Microsoft Hive ODBC Driver 2.01.09.1014

Apache Hive: 2.1.1-mapr-1901-r2

 
{quote}use strict;

use DBI;

use autodie;

 

my $dbh = DBI->connect(

    'DBI:ODBC:DSN=',

    ‘’,

    '',

    {

    RaiseError   => 1,

        PrintError   => 0,

    }

);

 

$dbh->do('set role admin;');
{quote}
 Running it produces the error which were in the OP logs
{quote}DBD::ODBC::db do failed: [Microsoft][Hardy] (61) Server returned error 
with no error message during operation: GetResultSetMetadata

TStatus.statusCode=ERROR_STATUS

    
TStatus.infoMessages="*java.lang.NullPointerException:null:16:15""org.apache.hive.service.cli.TableSchema::TableSchema.java:52""org.apache.hive.service.cli.TableSchema::TableSchema.java:59""org.apache.hive.service.cli.operation.SQLOperation:getResultSetSchema:SQLOperation.java:446""org.apache.hive.service.cli.operation.OperationManager:getOperationR
 (SQL-HY000)
{quote}
Perhaps the error occurs because "set role admin" doesn't return a resultset. 
Issuing regular SQL statements works fine

 

 

> ODBC - LOAD DATA - NullPointerException
> ---
>
> Key: HIVE-15814
> URL: https://issues.apache.org/jira/browse/HIVE-15814
> Project: Hive
>  Issue Type: Bug
>  Components: ODBC
>Affects Versions: 1.2.1
>Reporter: Pierre Villard
>Priority: Major
>
> Tested with :
> - Hive 1.2.1.2.4.0
> - Hive 1.2.1.2.5.3
> Information regarding ODBC access:
> - Aginity Workbench for Hadoop 
> - Version 4.8.0.2551 (build 6/14/2016) 
> - Microsoft Windows NT 6.1.7601 Service Pack 1 (64-bit) 
> - DBMS Version: 1.2.1000.2.5.3.0-37 Apache Hive 
> - Driver Used: Hive Native 
> - ODBC Driver Signature: {Aginity Hive}
> Request executed:
> {noformat}
> LOAD DATA INPATH '/tmp/AYX170201a327cc8faf8e562f5eb86225dd1fbf57' INTO TABLE 
> tmp.testpierre
> {noformat}
> Stacktrace available here:
> https://gist.github.com/pvillard31/9a08650f9a80ef0fa7e161d71c0d9e9b
> It seems to be specific to this driver shipped with Aginity studio. Same 
> tests with Simba ODBC driver or HWX ODBC driver are working fine.
> I suppose the NPE could be handled in a proper way even though it seems to be 
> related to some strange calls from ODBC driver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21698) TezSessionState#ensureLocalResources() causes IndexOutOfBoundsException while localizing resources

[Shubham Chaurasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shubham Chaurasia reassigned HIVE-21698:



> TezSessionState#ensureLocalResources() causes IndexOutOfBoundsException while 
> localizing resources
> --
>
> Key: HIVE-21698
> URL: https://issues.apache.org/jira/browse/HIVE-21698
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor, Tez
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>
> The cause of exception is due to [this loops 
> over|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L633]
>  {{newFilesNotFromConf}} and tries to access {{newResources.get(i)}} in that.
> Size of {{newResources}} is determined by [utils.localizeTempFiles() 
> here|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L629]
>  which may return a smaller list depending upon the {{skipFilesFromConf}} and 
> this would result in {{newResources.size()}} < {{newFilesNotFromConf.length}} 
> causing IndexOutOfBounds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21685) Wrong simplification in query with multiple IN clauses

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834365#comment-16834365
 ] 

Hive QA commented on HIVE-21685:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967952/HIVE-21685.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15981 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17143/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17143/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17143/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967952 - PreCommit-HIVE-Build

> Wrong simplification in query with multiple IN clauses
> --
>
> Key: HIVE-21685
> URL: https://issues.apache.org/jira/browse/HIVE-21685
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Oliver Draese
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21685.patch, HIVE-21685.patch, HIVE-21685.patch, 
> HIVE-21685.patch
>
>
> Simple test to reproduce:
> {code}
> select * from table1 where name IN(‘g’,‘r’) AND name IN(‘a’,‘b’);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21685) Wrong simplification in query with multiple IN clauses

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834346#comment-16834346
 ] 

Hive QA commented on HIVE-21685:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
1s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
43s{color} | {color:red} ql: The patch generated 3 new + 87 unchanged - 0 fixed 
= 90 total (was 87) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17143/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17143/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql itests U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17143/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Wrong simplification in query with multiple IN clauses
> --
>
> Key: HIVE-21685
> URL: https://issues.apache.org/jira/browse/HIVE-21685
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Oliver Draese
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21685.patch, HIVE-21685.patch, HIVE-21685.patch, 
> HIVE-21685.patch
>
>
> Simple test to reproduce:
> {code}
> select * from table1 where name IN(‘g’,‘r’) AND name IN(‘a’,‘b’);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21625) Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834337#comment-16834337
 ] 

Hive QA commented on HIVE-21625:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967953/HIVE-21625.4.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15981 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17142/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17142/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17142/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967953 - PreCommit-HIVE-Build

> Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method
> -
>
> Key: HIVE-21625
> URL: https://issues.apache.org/jira/browse/HIVE-21625
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21625.1.patch, HIVE-21625.2.patch, 
> HIVE-21625.3.patch, HIVE-21625.4.patch
>
>
> TxnIdUtils.checkEquivalentWriteIds has a bug which thinks (\{1,2,3,4\}, 6) 
> and (\{1,2,3,4,5,6\}, 8) compatible (the notation is (invalidlist, hwm)). 
> Here is a patch to fix it, also provide a comparison method to check which is 
> newer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21625) Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834312#comment-16834312
 ] 

Hive QA commented on HIVE-21625:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
27s{color} | {color:blue} storage-api in master has 48 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
12s{color} | {color:red} storage-api: The patch generated 63 new + 44 unchanged 
- 1 fixed = 107 total (was 45) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 22s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17142/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17142/yetus/diff-checkstyle-storage-api.txt
 |
| modules | C: storage-api U: storage-api |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17142/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method
> -
>
> Key: HIVE-21625
> URL: https://issues.apache.org/jira/browse/HIVE-21625
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21625.1.patch, HIVE-21625.2.patch, 
> HIVE-21625.3.patch, HIVE-21625.4.patch
>
>
> TxnIdUtils.checkEquivalentWriteIds has a bug which thinks (\{1,2,3,4\}, 6) 
> and (\{1,2,3,4,5,6\}, 8) compatible (the notation is (invalidlist, hwm)). 
> Here is a patch to fix it, also provide a comparison method to check which is 
> newer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21696) Include column stats in explain cbo formatted

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834306#comment-16834306
 ] 

Hive QA commented on HIVE-21696:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967951/HIVE-21696.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15980 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[concat_op] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input4] (batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[plan_json] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3] 
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join4] 
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6] 
(batchId=46)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17141/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17141/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17141/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967951 - PreCommit-HIVE-Build

> Include column stats in explain cbo formatted
> -
>
> Key: HIVE-21696
> URL: https://issues.apache.org/jira/browse/HIVE-21696
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21696.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21696) Include column stats in explain cbo formatted

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834298#comment-16834298
 ] 

Hive QA commented on HIVE-21696:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
3s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} ql: The patch generated 0 new + 0 unchanged - 1 
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 35s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17141/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17141/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Include column stats in explain cbo formatted
> -
>
> Key: HIVE-21696
> URL: https://issues.apache.org/jira/browse/HIVE-21696
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21696.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21697) Remove periodical full refresh in HMS cache

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-21697:
-


> Remove periodical full refresh in HMS cache
> ---
>
> Key: HIVE-21697
> URL: https://issues.apache.org/jira/browse/HIVE-21697
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
>
> In HIVE-18661, we added periodical notification based refresh in HMS cache. 
> We shall remove periodical full refresh to simplify the code as it will no 
> longer be used. In the mean time, we introduced mechanism to provide 
> monotonic reads through the CachedStore.commitTransaction. This will no 
> longer be needed after HIVE-21637. So I will remove related code as well. 
> This will provide some performance benefits include:
> 1. We don't have to slow down write to catch up notification logs. Write can 
> be done immediately and tag the cache with writeids
> 2. We can read from cache even if updateUsingNotificationEvents is running. 
> Read will compare the writeids of the cache so monotonic reads will be 
> guaranteed
> I'd like to put a patch separately with HIVE-21637 so it can be tested 
> independently. HMW will use periodical notification based refresh to update 
> cache. And it will temporarily lift the monotonic reads guarantee until 
> HIVE-21637 checkin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21152) Rewrite if expression to case and recognize simple case as an if

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834292#comment-16834292
 ] 

Hive QA commented on HIVE-21152:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967941/HIVE-21152.07.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 76 failed/errored test(s), 15980 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_when_case] 
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explode_null] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_case] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets_grouping]
 (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_join_preds] 
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[innerjoin1] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_outer] 
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_6] 
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_7] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_8] 
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_write_correct_definition_levels]
 (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf6] (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_if] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_offcbo] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_1] 
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_2] 
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce_3] 
(batchId=63)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constant_prop_when]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[kryo] 
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
 (batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
 (batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
 (batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
 (batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_case_when_1]
 (batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_case_when_2]
 (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_3]
 (batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping]
 (batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in] 
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin] 
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar] 
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select] 
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query11] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query21] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query31] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query34] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query36] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query47] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query4] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query53] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query57] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query63] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query67] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query70] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query73] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query74] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query78] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query86] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query89] 
(batchId=287)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query93] 
(batchId=287)

[jira] [Commented] (HIVE-21637) Synchronized metastore cache

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834274#comment-16834274
 ] 

Daniel Dai commented on HIVE-21637:
---

[~sankarh], can you take a look of the patch especially for writeid part? Note 
it is still WIP, but it's better to get initial feedback so I can make the 
changes earlier.

> Synchronized metastore cache
> 
>
> Key: HIVE-21637
> URL: https://issues.apache.org/jira/browse/HIVE-21637
> Project: Hive
>  Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21637-1.patch, HIVE-21637.2.patch, 
> HIVE-21637.3.patch, HIVE-21637.4.patch
>
>
> Currently, HMS has a cache implemented by CachedStore. The cache is 
> asynchronized and in HMS HA setting, we can only get eventual consistency. In 
> this Jira, we try to make it synchronized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21152) Rewrite if expression to case and recognize simple case as an if

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834268#comment-16834268
 ] 

Hive QA commented on HIVE-21152:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
4s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 53s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17140/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17140/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Rewrite if expression to case and recognize simple case as an if
> 
>
> Key: HIVE-21152
> URL: https://issues.apache.org/jira/browse/HIVE-21152
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21152.01.patch, HIVE-21152.02.patch, 
> HIVE-21152.03.patch, HIVE-21152.04.patch, HIVE-21152.05.patch, 
> HIVE-21152.06.patch, HIVE-21152.07.patch
>
>
> * {{IF}} is not part of the sql standard; however given its special form its 
> simpler - and currently in Hive it also has vectorized support
> * people writing standard sql may write: {{CASE WHEN member=1 THEN attr+1 
> else attr+2 END}} which is essentially an if.
> The idea is to rewrite IFs to CASEs for the cbo; and recognize simple 
> "CASE"-s as IFs to get vectorization on them if possible



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19353) Vectorization: ConstantVectorExpression --> RuntimeException: Unexpected column vector type LIST

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834254#comment-16834254
 ] 

Vineet Garg commented on HIVE-19353:


+1

> Vectorization: ConstantVectorExpression  --> RuntimeException: Unexpected 
> column vector type LIST
> -
>
> Key: HIVE-19353
> URL: https://issues.apache.org/jira/browse/HIVE-19353
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19353.01.patch, HIVE-19353.02.patch, 
> HIVE-19353.03.patch, HIVE-19353.04.patch
>
>
> Found by enabling vectorization for 
> org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
> {noformat}
> Caused by: java.lang.RuntimeException: Unexpected column vector type LIST
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression.evaluate(ConstantVectorExpression.java:237)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.closeOp(VectorMapOperator.java:984)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834259#comment-16834259
 ] 

Vineet Garg edited comment on HIVE-21690 at 5/6/19 11:01 PM:
-

bq. Why should we consider only aggregate operators? What about other operators?
One of the reason for considering aggregate operator beside join operator is 
that similar to join operator it involves (in most of the cases) shuffling data 
and therefore it adds significant cost to the overall execution unlike other 
operators.

bq. Since cost model is pluggable, have you thought about creating a cost model 
that extends the join reordering (default) one with cost calculation for the 
Aggregate operator? You could use the new cost model when you trigger this 
rule. In a follow-up, you can study whether using the same cost model for join 
reordering makes sense or not, and evaluate the merit of that change for join 
reordering on its own.
I really like this suggestion and I think this is better approach. Like you 
suggested it is prudent to  evaluate the change in cost model before making it 
permanent.


was (Author: vgarg):
bq. Why should we consider only aggregate operators? What about other operators?
One of the reason for considering aggregate operator beside join operator is 
that similar to join operator it involves (in most of the cases) shuffling data 
and therefore it adds significant cost to the overall execution unlike other 
operators.

bq. Since cost model is pluggable, have you thought about creating a cost model 
that extends the join reordering (default) one with cost calculation for the 
Aggregate operator? You could use the new cost model when you trigger this 
rule. In a follow-up, you can study whether using the same cost model for join 
reordering makes sense or not, and evaluate the merit of that change for join 
reordering on its own.
I really like this suggestion and I think this is better approach. Like you 
suggested it is prudent to  evaluate the change in cost model change before 
making it permanent.

> Support outer joins with HiveAggregateJoinTransposeRule and turn it on by 
> default
> -
>
> Key: HIVE-21690
> URL: https://issues.apache.org/jira/browse/HIVE-21690
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21690.1.patch
>
>
> 1) This optimization is off by default. We would like to turn on this 
> optimization wherein group by is pushed down to join, in some cases top 
> aggregate is removed but in most of the cases this optimization adds extra 
> aggregate nodes. To measure if those extra aggregates are beneficial or not 
> (they might add extra overhead without reducing rows) cost is computed and 
> compared b/w previous plan and new plan.
> Since Hive's cost model only consider JOIN's cost and discard cost of rest of 
> the nodes, this comparison always favor new plan (since adding aggregate 
> beneath join reduces the total number of rows processed by the join and 
> therefore reduces the join cost). Therefore turning on this optimization with 
> existing cost model is not a good idea.
> One approach to fix this is to localize the cost computation to the rule 
> itself, i.e compute the non-cumulative cost of existing aggregate and join 
> and compare it with new cost of new aggregates, join and top aggregate.
> Better approach in my opinion would be to fix the cost model and take 
> aggregate cost into account (along with the join). This could affect other 
> queries and can cause performance regression but those will most likely be 
> issues with the planning and should be investigated and fixed.
> 2) This optimization currently only support INNER JOIN. This can be extended 
> to support OUTER joins.
>  
> cc [~jcamachorodriguez] [~ashutoshc] [~gopalv]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834259#comment-16834259
 ] 

Vineet Garg commented on HIVE-21690:


bq. Why should we consider only aggregate operators? What about other operators?
One of the reason for considering aggregate operator beside join operator is 
that similar to join operator it involves (in most of the cases) shuffling data 
and therefore it adds significant cost to the overall execution unlike other 
operators.

bq. Since cost model is pluggable, have you thought about creating a cost model 
that extends the join reordering (default) one with cost calculation for the 
Aggregate operator? You could use the new cost model when you trigger this 
rule. In a follow-up, you can study whether using the same cost model for join 
reordering makes sense or not, and evaluate the merit of that change for join 
reordering on its own.
I really like this suggestion and I think this is better approach. Like you 
suggested it is prudent to  evaluate the change in cost model change before 
making it permanent.

> Support outer joins with HiveAggregateJoinTransposeRule and turn it on by 
> default
> -
>
> Key: HIVE-21690
> URL: https://issues.apache.org/jira/browse/HIVE-21690
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21690.1.patch
>
>
> 1) This optimization is off by default. We would like to turn on this 
> optimization wherein group by is pushed down to join, in some cases top 
> aggregate is removed but in most of the cases this optimization adds extra 
> aggregate nodes. To measure if those extra aggregates are beneficial or not 
> (they might add extra overhead without reducing rows) cost is computed and 
> compared b/w previous plan and new plan.
> Since Hive's cost model only consider JOIN's cost and discard cost of rest of 
> the nodes, this comparison always favor new plan (since adding aggregate 
> beneath join reduces the total number of rows processed by the join and 
> therefore reduces the join cost). Therefore turning on this optimization with 
> existing cost model is not a good idea.
> One approach to fix this is to localize the cost computation to the rule 
> itself, i.e compute the non-cumulative cost of existing aggregate and join 
> and compare it with new cost of new aggregates, join and top aggregate.
> Better approach in my opinion would be to fix the cost model and take 
> aggregate cost into account (along with the join). This could affect other 
> queries and can cause performance regression but those will most likely be 
> issues with the planning and should be investigated and fixed.
> 2) This optimization currently only support INNER JOIN. This can be extended 
> to support OUTER joins.
>  
> cc [~jcamachorodriguez] [~ashutoshc] [~gopalv]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834241#comment-16834241
 ] 

Jesus Camacho Rodriguez edited comment on HIVE-21690 at 5/6/19 10:57 PM:
-

{quote}
One approach to fix this is to localize the cost computation to the rule 
itself, i.e compute the non-cumulative cost of existing aggregate and join and 
compare it with new cost of new aggregates, join and top aggregate.
Better approach in my opinion would be to fix the cost model and take aggregate 
cost into account (along with the join). This could affect other queries and 
can cause performance regression but those will most likely be issues with the 
planning and should be investigated and fixed.
{quote}
In principle, second approach seems the logical choice since it takes the cost 
model closer to execution actual cost. However, it can easily backfire with 
current implementation of join reordering costing, which only considers cost 
for join operation and builds on that assumption. Why should we consider only 
aggregate operators? What about other operators?
Before pushing such a change, I would argue that we need further evaluation on 
how this will affect join reordering and the regressions that we will get wrt 
previous benchmarks.

Since cost model is pluggable, have you thought about creating a cost model 
that extends the join reordering (default) one with cost calculation for the 
Aggregate operator? You could use the new cost model when you trigger this 
rule. In a follow-up, you can study whether using the same cost model for join 
reordering makes sense or not, and evaluate the merit of that change for join 
reordering on its own.


was (Author: jcamachorodriguez):
{quote}
One approach to fix this is to localize the cost computation to the rule 
itself, i.e compute the non-cumulative cost of existing aggregate and join and 
compare it with new cost of new aggregates, join and top aggregate.
Better approach in my opinion would be to fix the cost model and take aggregate 
cost into account (along with the join). This could affect other queries and 
can cause performance regression but those will most likely be issues with the 
planning and should be investigated and fixed.
{quote}
In principle, second approach seems the logical choice since it takes the cost 
model closer to execution actual cost. However, it can easily backfire with 
current implementation of join reordering costing, which only considers cost 
for join operation and builds on that assumption. Why should we consider only 
aggregate operators? What about other operators?
Before pushing such a change, I would argue that we need further evaluation on 
how this will affect join reordering and the regressions that we will get wrt 
previous benchmarks.

Since cost model is pluggable, have you thought about creating a cost model 
that extends the join reordering (default) one with cost calculation for the 
Aggregate operator? You could use the new cost model when you trigger this 
rule. In a follow-up, you can study whether using the same cost model for join 
reordering makes sense or not, and evaluate the merit of that change for join 
reordering on its own.

> Support outer joins with HiveAggregateJoinTransposeRule and turn it on by 
> default
> -
>
> Key: HIVE-21690
> URL: https://issues.apache.org/jira/browse/HIVE-21690
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21690.1.patch
>
>
> 1) This optimization is off by default. We would like to turn on this 
> optimization wherein group by is pushed down to join, in some cases top 
> aggregate is removed but in most of the cases this optimization adds extra 
> aggregate nodes. To measure if those extra aggregates are beneficial or not 
> (they might add extra overhead without reducing rows) cost is computed and 
> compared b/w previous plan and new plan.
> Since Hive's cost model only consider JOIN's cost and discard cost of rest of 
> the nodes, this comparison always favor new plan (since adding aggregate 
> beneath join reduces the total number of rows processed by the join and 
> therefore reduces the join cost). Therefore turning on this optimization with 
> existing cost model is not a good idea.
> One approach to fix this is to localize the cost computation to the rule 
> itself, i.e compute the non-cumulative cost of existing aggregate and join 
> and compare it with new cost of new aggregates, join and top aggregate.
> Better approach in my opinion would be to fix the cost model and take 
> aggregate cost into account (along with the join). This could affect other 
> 

[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834252#comment-16834252
 ] 

Hive QA commented on HIVE-21304:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967940/HIVE-21304.05.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 15980 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
 (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_buckets] (batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp] (batchId=32)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_12]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sharedwork] 
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
 (batchId=192)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_12]
 (batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket4] 
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7] 
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[disable_merge_for_bucketing]
 (batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join32_lessSize] 
(batchId=115)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17139/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17139/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17139/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967940 - PreCommit-HIVE-Build

> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, 
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13582?focusedWorklogId=238184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238184
 ]

ASF GitHub Bot logged work on HIVE-13582:
-

Author: ASF GitHub Bot
Created on: 06/May/19 22:44
Start Date: 06/May/19 22:44
Worklog Time Spent: 10m 
  Work Description: jcamachor commented on pull request #612: HIVE-13582: 
Support for Quantified predicates for non-correlated queries
URL: https://github.com/apache/hive/pull/612#discussion_r281394026
 
 

 ##
 File path: ql/src/test/results/clientpositive/llap/subquery_ALL.q.out
 ##
 @@ -0,0 +1,542 @@
+PREHOOK: query: create table tempty(i int, j int)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tempty
+POSTHOOK: query: create table tempty(i int, j int)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tempty
+PREHOOK: query: CREATE TABLE part_null_n0 as select * from part
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@part
+PREHOOK: Output: database:default
+PREHOOK: Output: default@part_null_n0
+POSTHOOK: query: CREATE TABLE part_null_n0 as select * from part
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@part
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@part_null_n0
+POSTHOOK: Lineage: part_null_n0.p_brand SIMPLE 
[(part)part.FieldSchema(name:p_brand, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_comment SIMPLE 
[(part)part.FieldSchema(name:p_comment, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_container SIMPLE 
[(part)part.FieldSchema(name:p_container, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_mfgr SIMPLE 
[(part)part.FieldSchema(name:p_mfgr, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_name SIMPLE 
[(part)part.FieldSchema(name:p_name, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_partkey SIMPLE 
[(part)part.FieldSchema(name:p_partkey, type:int, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_retailprice SIMPLE 
[(part)part.FieldSchema(name:p_retailprice, type:double, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_size SIMPLE 
[(part)part.FieldSchema(name:p_size, type:int, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_type SIMPLE 
[(part)part.FieldSchema(name:p_type, type:string, comment:null), ]
+PREHOOK: query: insert into part_null_n0 values(NULL,NULL,NULL,NULL,NULL, 
NULL, NULL,NULL,NULL)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@part_null_n0
+POSTHOOK: query: insert into part_null_n0 values(NULL,NULL,NULL,NULL,NULL, 
NULL, NULL,NULL,NULL)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@part_null_n0
+POSTHOOK: Lineage: part_null_n0.p_brand EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_comment EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_container EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_mfgr EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_name EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_partkey EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_retailprice EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_size EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_type EXPRESSION []
+Warning: Shuffle Join MERGEJOIN[37][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in 
Stage 'Reducer 3' is a cross product
+PREHOOK: query: explain cbo select count(*) from part where p_partkey <> ALL 
(select p_partkey from part)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@part
+ A masked pattern was here 
+POSTHOOK: query: explain cbo select count(*) from part where p_partkey <> ALL 
(select p_partkey from part)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@part
+ A masked pattern was here 
+CBO PLAN:
+HiveAggregate(group=[{}], agg#0=[count()])
+  HiveFilter(condition=[AND(OR(IS NULL($4), =($1, 0)), OR(IS NOT NULL($0), 
=($1, 0), IS NOT NULL($4)), OR(>=($2, $1), =($1, 0), IS NOT NULL($4), IS 
NULL($0)))])
 
 Review comment:
   Yes, it makes sense... This may be a problem with other queries too, since 
algorithm ends up reordering joins considering that one of them is a cartesian 
product (but it is not).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 238184)
Time Spent: 2h  (was: 1h 50m)

> E061-07 and E061-12: Quantified Comparison Predicates
> -
>
> Key: HIVE-13582
> URL: 

[jira] [Commented] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834241#comment-16834241
 ] 

Jesus Camacho Rodriguez commented on HIVE-21690:


{quote}
One approach to fix this is to localize the cost computation to the rule 
itself, i.e compute the non-cumulative cost of existing aggregate and join and 
compare it with new cost of new aggregates, join and top aggregate.
Better approach in my opinion would be to fix the cost model and take aggregate 
cost into account (along with the join). This could affect other queries and 
can cause performance regression but those will most likely be issues with the 
planning and should be investigated and fixed.
{quote}
In principle, second approach seems the logical choice since it takes the cost 
model closer to execution actual cost. However, it can easily backfire with 
current implementation of join reordering costing, which only considers cost 
for join operation and builds on that assumption. Why should we consider only 
aggregate operators? What about other operators?
Before pushing such a change, I would argue that we need further evaluation on 
how this will affect join reordering and the regressions that we will get wrt 
previous benchmarks.

Since cost model is pluggable, have you thought about creating a cost model 
that extends the join reordering (default) one with cost calculation for the 
Aggregate operator? You could use the new cost model when you trigger this 
rule. In a follow-up, you can study whether using the same cost model for join 
reordering makes sense or not, and evaluate the merit of that change for join 
reordering on its own.

> Support outer joins with HiveAggregateJoinTransposeRule and turn it on by 
> default
> -
>
> Key: HIVE-21690
> URL: https://issues.apache.org/jira/browse/HIVE-21690
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21690.1.patch
>
>
> 1) This optimization is off by default. We would like to turn on this 
> optimization wherein group by is pushed down to join, in some cases top 
> aggregate is removed but in most of the cases this optimization adds extra 
> aggregate nodes. To measure if those extra aggregates are beneficial or not 
> (they might add extra overhead without reducing rows) cost is computed and 
> compared b/w previous plan and new plan.
> Since Hive's cost model only consider JOIN's cost and discard cost of rest of 
> the nodes, this comparison always favor new plan (since adding aggregate 
> beneath join reduces the total number of rows processed by the join and 
> therefore reduces the join cost). Therefore turning on this optimization with 
> existing cost model is not a good idea.
> One approach to fix this is to localize the cost computation to the rule 
> itself, i.e compute the non-cumulative cost of existing aggregate and join 
> and compare it with new cost of new aggregates, join and top aggregate.
> Better approach in my opinion would be to fix the cost model and take 
> aggregate cost into account (along with the join). This could affect other 
> queries and can cause performance regression but those will most likely be 
> issues with the planning and should be investigated and fixed.
> 2) This optimization currently only support INNER JOIN. This can be extended 
> to support OUTER joins.
>  
> cc [~jcamachorodriguez] [~ashutoshc] [~gopalv]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834232#comment-16834232
 ] 

Vineet Garg commented on HIVE-21304:


[~kgyrtkirk] Can you create pull request/review request?

> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, 
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21656) Vectorize UDF mask

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834230#comment-16834230
 ] 

Vineet Garg commented on HIVE-21656:


[~gopalv] I have addressed the review comments and have updated the pull 
request. Can you take a look?

> Vectorize UDF mask
> --
>
> Key: HIVE-21656
> URL: https://issues.apache.org/jira/browse/HIVE-21656
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21656.1.patch, HIVE-21656.2.patch, 
> HIVE-21656.3.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-21690:
---
Description: 
1) This optimization is off by default. We would like to turn on this 
optimization wherein group by is pushed down to join, in some cases top 
aggregate is removed but in most of the cases this optimization adds extra 
aggregate nodes. To measure if those extra aggregates are beneficial or not 
(they might add extra overhead without reducing rows) cost is computed and 
compared b/w previous plan and new plan.

Since Hive's cost model only consider JOIN's cost and discard cost of rest of 
the nodes, this comparison always favor new plan (since adding aggregate 
beneath join reduces the total number of rows processed by the join and 
therefore reduces the join cost). Therefore turning on this optimization with 
existing cost model is not a good idea.

One approach to fix this is to localize the cost computation to the rule 
itself, i.e compute the non-cumulative cost of existing aggregate and join and 
compare it with new cost of new aggregates, join and top aggregate.

Better approach in my opinion would be to fix the cost model and take aggregate 
cost into account (along with the join). This could affect other queries and 
can cause performance regression but those will most likely be issues with the 
planning and should be investigated and fixed.

2) This optimization currently only support INNER JOIN. This can be extended to 
support OUTER joins.

 

cc [~jcamachorodriguez] [~ashutoshc] [~gopalv]

  was:
1) This optimization is off by default. We would like to turn on this 
optimization wherein group by is pushed down to join, in some cases top 
aggregate is removed but in most of the cases this optimization adds extra 
aggregate nodes. To measure if those extra aggregates are beneficial or not 
(they might add extra overhead without reducing rows) cost is computed and 
compared b/w previous plan and new plan.

Since Hive's cost model only consider JOIN's cost and discard cost of rest of 
the nodes, this comparison always favor new plan (since adding aggregate 
beneath join reduces the total number of rows processed by the join and 
therefore reduces the join cost). Therefore turning on this optimization with 
existing cost model is not a good idea.

One approach to fix this is to localize the cost computation to the rule 
itself, i.e compute the non-cumulative cost of existing aggregate and join and 
compare it with new cost of new aggregates, join and top aggregate. 

Better approach in my opinion would be to fix the cost model and take aggregate 
cost into account (along with the join). This could affect other queries and 
can cause performance regression but those will most likely be issues with the 
planning and should be investigated and fixed.


2) This optimization currently only support INNER JOIN. This can be extended to 
support OUTER joins.


> Support outer joins with HiveAggregateJoinTransposeRule and turn it on by 
> default
> -
>
> Key: HIVE-21690
> URL: https://issues.apache.org/jira/browse/HIVE-21690
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21690.1.patch
>
>
> 1) This optimization is off by default. We would like to turn on this 
> optimization wherein group by is pushed down to join, in some cases top 
> aggregate is removed but in most of the cases this optimization adds extra 
> aggregate nodes. To measure if those extra aggregates are beneficial or not 
> (they might add extra overhead without reducing rows) cost is computed and 
> compared b/w previous plan and new plan.
> Since Hive's cost model only consider JOIN's cost and discard cost of rest of 
> the nodes, this comparison always favor new plan (since adding aggregate 
> beneath join reduces the total number of rows processed by the join and 
> therefore reduces the join cost). Therefore turning on this optimization with 
> existing cost model is not a good idea.
> One approach to fix this is to localize the cost computation to the rule 
> itself, i.e compute the non-cumulative cost of existing aggregate and join 
> and compare it with new cost of new aggregates, join and top aggregate.
> Better approach in my opinion would be to fix the cost model and take 
> aggregate cost into account (along with the join). This could affect other 
> queries and can cause performance regression but those will most likely be 
> issues with the planning and should be investigated and fixed.
> 2) This optimization currently only support INNER JOIN. This can be extended 
> to support OUTER joins.
>  
> cc [~jcamachorodriguez] [~ashutoshc] 

[jira] [Work logged] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13582?focusedWorklogId=238168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238168
 ]

ASF GitHub Bot logged work on HIVE-13582:
-

Author: ASF GitHub Bot
Created on: 06/May/19 22:06
Start Date: 06/May/19 22:06
Worklog Time Spent: 10m 
  Work Description: vineetgarg02 commented on pull request #612: 
HIVE-13582: Support for Quantified predicates for non-correlated queries
URL: https://github.com/apache/hive/pull/612#discussion_r281381665
 
 

 ##
 File path: ql/src/test/results/clientpositive/llap/subquery_ALL.q.out
 ##
 @@ -0,0 +1,542 @@
+PREHOOK: query: create table tempty(i int, j int)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tempty
+POSTHOOK: query: create table tempty(i int, j int)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tempty
+PREHOOK: query: CREATE TABLE part_null_n0 as select * from part
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@part
+PREHOOK: Output: database:default
+PREHOOK: Output: default@part_null_n0
+POSTHOOK: query: CREATE TABLE part_null_n0 as select * from part
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@part
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@part_null_n0
+POSTHOOK: Lineage: part_null_n0.p_brand SIMPLE 
[(part)part.FieldSchema(name:p_brand, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_comment SIMPLE 
[(part)part.FieldSchema(name:p_comment, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_container SIMPLE 
[(part)part.FieldSchema(name:p_container, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_mfgr SIMPLE 
[(part)part.FieldSchema(name:p_mfgr, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_name SIMPLE 
[(part)part.FieldSchema(name:p_name, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_partkey SIMPLE 
[(part)part.FieldSchema(name:p_partkey, type:int, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_retailprice SIMPLE 
[(part)part.FieldSchema(name:p_retailprice, type:double, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_size SIMPLE 
[(part)part.FieldSchema(name:p_size, type:int, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_type SIMPLE 
[(part)part.FieldSchema(name:p_type, type:string, comment:null), ]
+PREHOOK: query: insert into part_null_n0 values(NULL,NULL,NULL,NULL,NULL, 
NULL, NULL,NULL,NULL)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@part_null_n0
+POSTHOOK: query: insert into part_null_n0 values(NULL,NULL,NULL,NULL,NULL, 
NULL, NULL,NULL,NULL)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@part_null_n0
+POSTHOOK: Lineage: part_null_n0.p_brand EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_comment EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_container EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_mfgr EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_name EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_partkey EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_retailprice EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_size EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_type EXPRESSION []
+Warning: Shuffle Join MERGEJOIN[37][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in 
Stage 'Reducer 3' is a cross product
+PREHOOK: query: explain cbo select count(*) from part where p_partkey <> ALL 
(select p_partkey from part)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@part
+ A masked pattern was here 
+POSTHOOK: query: explain cbo select count(*) from part where p_partkey <> ALL 
(select p_partkey from part)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@part
+ A masked pattern was here 
+CBO PLAN:
+HiveAggregate(group=[{}], agg#0=[count()])
+  HiveFilter(condition=[AND(OR(IS NULL($4), =($1, 0)), OR(IS NOT NULL($0), 
=($1, 0), IS NOT NULL($4)), OR(>=($2, $1), =($1, 0), IS NOT NULL($4), IS 
NULL($0)))])
 
 Review comment:
   It looks like HiveFilterJoinRule isn't able to push the filter on LEFT join 
(before join re-ordering FILTER is on top of LEFT JOIN instead of on INNER 
JOIN).
   I can create a follow-up to further investigate this and may be fix/improve 
it if possible. Let me know what do you think.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 238168)
Time Spent: 1h 50m  (was: 1h 40m)

> E061-07 and E061-12: Quantified Comparison Predicates
> 

[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834228#comment-16834228
 ] 

Hive QA commented on HIVE-21304:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
1s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
52s{color} | {color:red} ql: The patch generated 2 new + 997 unchanged - 3 
fixed = 999 total (was 1000) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 28s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17139/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17139/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql itests/hive-blobstore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17139/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, 
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19353) Vectorization: ConstantVectorExpression --> RuntimeException: Unexpected column vector type LIST

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834210#comment-16834210
 ] 

Hive QA commented on HIVE-19353:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967939/HIVE-19353.04.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15981 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17138/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17138/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17138/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967939 - PreCommit-HIVE-Build

> Vectorization: ConstantVectorExpression  --> RuntimeException: Unexpected 
> column vector type LIST
> -
>
> Key: HIVE-19353
> URL: https://issues.apache.org/jira/browse/HIVE-19353
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19353.01.patch, HIVE-19353.02.patch, 
> HIVE-19353.03.patch, HIVE-19353.04.patch
>
>
> Found by enabling vectorization for 
> org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
> {noformat}
> Caused by: java.lang.RuntimeException: Unexpected column vector type LIST
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression.evaluate(ConstantVectorExpression.java:237)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.closeOp(VectorMapOperator.java:984)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19353) Vectorization: ConstantVectorExpression --> RuntimeException: Unexpected column vector type LIST

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834173#comment-16834173
 ] 

Hive QA commented on HIVE-19353:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
1s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
43s{color} | {color:red} ql: The patch generated 1 new + 375 unchanged - 0 
fixed = 376 total (was 375) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17138/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17138/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17138/yetus/whitespace-eol.txt
 |
| modules | C: ql itests U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17138/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Vectorization: ConstantVectorExpression  --> RuntimeException: Unexpected 
> column vector type LIST
> -
>
> Key: HIVE-19353
> URL: https://issues.apache.org/jira/browse/HIVE-19353
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19353.01.patch, HIVE-19353.02.patch, 
> HIVE-19353.03.patch, HIVE-19353.04.patch
>
>
> Found by enabling vectorization for 
> org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
> {noformat}
> Caused by: java.lang.RuntimeException: Unexpected column vector type LIST
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression.evaluate(ConstantVectorExpression.java:237)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> 

[jira] [Commented] (HIVE-21576) Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834157#comment-16834157
 ] 

Hive QA commented on HIVE-21576:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967936/HIVE-21576.2.patch

{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 16004 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[udf_unix_timestamp] 
(batchId=285)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sql_formats] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_add_months] 
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_date_format] 
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_from_unixtime] 
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_string] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_to_unix_timestamp] 
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_unix_timestamp] 
(batchId=15)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFCastWithFormat.testDateToStringWithFormat
 (batchId=304)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFCastWithFormat.testStringToDateWithFormat
 (batchId=304)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFCastWithFormat.testTimestampTZToStringWithFormat
 (batchId=304)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFCastWithFormat.testTimestampToStringWithFormat
 (batchId=304)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17137/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17137/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17137/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967936 - PreCommit-HIVE-Build

> Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats
> -
>
> Key: HIVE-21576
> URL: https://issues.apache.org/jira/browse/HIVE-21576
> Project: Hive
>  Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21576.01.patch, HIVE-21576.2.patch
>
>
> Introduce FORMAT clause to CAST statements as well as the below limited list 
> of SQL:2016 datetime formats to Hive in general. These can be used if a 
> session-level feature flag is turned on.
>  * 
>  * MM
>  * DD
>  * HH
>  * MI
>  * SS
> Definitions of these formats here: 
> [https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21637) Synchronized metastore cache

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21637:
--
Attachment: HIVE-21637.4.patch

> Synchronized metastore cache
> 
>
> Key: HIVE-21637
> URL: https://issues.apache.org/jira/browse/HIVE-21637
> Project: Hive
>  Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21637-1.patch, HIVE-21637.2.patch, 
> HIVE-21637.3.patch, HIVE-21637.4.patch
>
>
> Currently, HMS has a cache implemented by CachedStore. The cache is 
> asynchronized and in HMS HA setting, we can only get eventual consistency. In 
> this Jira, we try to make it synchronized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-14888) SparkClientImpl checks for "kerberos" string in hiveconf only when determining whether to use keytab file.

[David McGinnis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David McGinnis updated HIVE-14888:
--
Status: In Progress  (was: Patch Available)

> SparkClientImpl checks for "kerberos" string in hiveconf only when 
> determining whether to use keytab file.
> --
>
> Key: HIVE-14888
> URL: https://issues.apache.org/jira/browse/HIVE-14888
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Thomas Rega
>Assignee: David McGinnis
>Priority: Major
> Attachments: HIVE-14888.1-spark.patch, HIVE-14888.2.patch, 
> HIVE-14888.3.patch, HIVE-14888.4.patch
>
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> The SparkClientImpl will only provide a principal and keytab argument if the 
> HADOOP_SECURITY_AUTHENTICATION in hive conf is set to "kerberos". This will 
> not work on clusters with Hadoop security enabled that are not configured as 
> "kerberos", for example, a cluster which is configured for "ldap".
> The solution is to call UserGroupInformation.isSecurityEnabled() instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21576) Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834138#comment-16834138
 ] 

Hive QA commented on HIVE-21576:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
34s{color} | {color:blue} common in master has 62 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
41s{color} | {color:blue} serde in master has 197 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
59s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} common: The patch generated 18 new + 439 unchanged - 0 
fixed = 457 total (was 439) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} serde: The patch generated 3 new + 222 unchanged - 1 
fixed = 225 total (was 223) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
42s{color} | {color:red} ql: The patch generated 37 new + 542 unchanged - 21 
fixed = 579 total (was 563) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
11s{color} | {color:red} ql generated 28 new + 2226 unchanged - 27 fixed = 2254 
total (was 2253) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 10s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Found reliance on default encoding in 
org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToStringWithFormat.func(BytesColumnVector,
 long[], int):in 
org.apache.hadoop.hive.ql.exec.vector.expressions.CastDateToStringWithFormat.func(BytesColumnVector,
 long[], int): String.getBytes()  At CastDateToStringWithFormat.java:[line 65] |
|  |  Found reliance on default encoding in 
org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.func(BytesColumnVector,
 TimestampColumnVector, int, HiveDateTimeFormatter):in 
org.apache.hadoop.hive.ql.exec.vector.expressions.CastTimestampToString.func(BytesColumnVector,
 TimestampColumnVector, int, HiveDateTimeFormatter): String.getBytes()  At 
CastTimestampToString.java:[line 84] |
|  |  Dead store to KW_IN243 in 
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceSimilarExpressionAtom(CommonTree)
  At 
HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceSimilarExpressionAtom(CommonTree)
  At HiveParser_IdentifiersParser.java:[line 9631] |
|  |  Dead store to LPAREN233 in 
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.subQueryExpression()
  At 

[jira] [Updated] (HIVE-14888) SparkClientImpl checks for "kerberos" string in hiveconf only when determining whether to use keytab file.

[David McGinnis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David McGinnis updated HIVE-14888:
--
Fix Version/s: 4.0.0
   Attachment: HIVE-14888.5.patch
   Status: Patch Available  (was: In Progress)

> SparkClientImpl checks for "kerberos" string in hiveconf only when 
> determining whether to use keytab file.
> --
>
> Key: HIVE-14888
> URL: https://issues.apache.org/jira/browse/HIVE-14888
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Thomas Rega
>Assignee: David McGinnis
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-14888.1-spark.patch, HIVE-14888.2.patch, 
> HIVE-14888.3.patch, HIVE-14888.4.patch, HIVE-14888.5.patch
>
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> The SparkClientImpl will only provide a principal and keytab argument if the 
> HADOOP_SECURITY_AUTHENTICATION in hive conf is set to "kerberos". This will 
> not work on clusters with Hadoop security enabled that are not configured as 
> "kerberos", for example, a cluster which is configured for "ldap".
> The solution is to call UserGroupInformation.isSecurityEnabled() instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21679) Replicating a CTAS event creating an MM table fails.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-21679:

Description: 
use dumpdb;
create table t1 (a int, b int);
insert into t1 values (1, 2), (3, 4);
create table t6_mm_part partitioned by (a) stored as orc tblproperties 
("transactional"="true", "transactional_properties"="insert_only") as select * 
from t1
create table t6_mm stored as orc tblproperties ("transactional"="true", 
"transactional_properties"="insert_only") as select * from t1;
repl dump dumpdb;
create table t6_mm_part_2 partitioned by (a) stored as orc tblproperties 
("transactional"="true", "transactional_properties"="insert_only") as select * 
from t1;
create table t6_mm_2 partitioned by (a) stored as orc tblproperties 
("transactional"="true", "transactional_properties"="insert_only") as select * 
from t1;
repl dump dumpdb from 
repl load loaddb from '/tmp/dump/next';

{code}
ERROR : failed replication
org.apache.hadoop.hive.ql.parse.SemanticException: Invalid table name 
loaddb.dumpdb.t6_mm_part_2
 at 
org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2253) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2239) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hadoop.hive.ql.plan.AlterTableDesc.setOldName(AlterTableDesc.java:419)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.tableUpdateReplStateTask(IncrementalLoadTasksBuilder.java:286)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.addUpdateReplStateTasks(IncrementalLoadTasksBuilder.java:371)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:244)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(IncrementalLoadTasksBuilder.java:139)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.executeIncrementalLoad(ReplLoadTask.java:488)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.execute(ReplLoadTask.java:102) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:88)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:332)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
 at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
 ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
 at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:350)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_191]
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_191]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
~[?:1.8.0_191]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
~[?:1.8.0_191]
 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
ERROR : FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask. Invalid table name 
loaddb.dumpdb.t6_mm_part_2
Error: Error while processing statement: FAILED: Execution Error, return code 1 
from 

[jira] [Updated] (HIVE-21679) Replicating a CTAS event creating an MM table fails.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-21679:
--
Labels: CTAS pull-request-available  (was: CTAS)

> Replicating a CTAS event creating an MM table fails.
> 
>
> Key: HIVE-21679
> URL: https://issues.apache.org/jira/browse/HIVE-21679
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: CTAS, pull-request-available
> Attachments: HIVE-21679.01.patch
>
>
> use dumpdb;
> create table t1 (a int, b int);
> insert into t1 values (1, 2), (3, 4);
> create table t6_mm_part partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1
> create table t6_mm stored as orc tblproperties ("transactional"="true", 
> "transactional_properties"="insert_only") as select * from t1;
> repl dump dumpdb;
> create table t6_mm_part_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> create table t6_mm_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> repl dump dumpdb from 
> repl load loaddb from '/tmp/dump/next';
> ERROR : failed replication
> org.apache.hadoop.hive.ql.parse.SemanticException: Invalid table name 
> loaddb.dumpdb.t6_mm_part_2
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2253) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2239) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.plan.AlterTableDesc.setOldName(AlterTableDesc.java:419)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.tableUpdateReplStateTask(IncrementalLoadTasksBuilder.java:286)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.addUpdateReplStateTasks(IncrementalLoadTasksBuilder.java:371)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:244)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(IncrementalLoadTasksBuilder.java:139)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.executeIncrementalLoad(ReplLoadTask.java:488)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.execute(ReplLoadTask.java:102)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:88)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:332)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
>  at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
>  ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
>  at 
> 

[jira] [Work logged] (HIVE-21679) Replicating a CTAS event creating an MM table fails.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21679?focusedWorklogId=238024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238024
 ]

ASF GitHub Bot logged work on HIVE-21679:
-

Author: ASF GitHub Bot
Created on: 06/May/19 19:13
Start Date: 06/May/19 19:13
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #617: HIVE-21679: 
Replicating a CTAS event creating an MM table fails.
URL: https://github.com/apache/hive/pull/617
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 238024)
Time Spent: 10m
Remaining Estimate: 0h

> Replicating a CTAS event creating an MM table fails.
> 
>
> Key: HIVE-21679
> URL: https://issues.apache.org/jira/browse/HIVE-21679
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: CTAS, pull-request-available
> Attachments: HIVE-21679.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> use dumpdb;
> create table t1 (a int, b int);
> insert into t1 values (1, 2), (3, 4);
> create table t6_mm_part partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1
> create table t6_mm stored as orc tblproperties ("transactional"="true", 
> "transactional_properties"="insert_only") as select * from t1;
> repl dump dumpdb;
> create table t6_mm_part_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> create table t6_mm_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> repl dump dumpdb from 
> repl load loaddb from '/tmp/dump/next';
> ERROR : failed replication
> org.apache.hadoop.hive.ql.parse.SemanticException: Invalid table name 
> loaddb.dumpdb.t6_mm_part_2
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2253) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2239) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.plan.AlterTableDesc.setOldName(AlterTableDesc.java:419)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.tableUpdateReplStateTask(IncrementalLoadTasksBuilder.java:286)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.addUpdateReplStateTasks(IncrementalLoadTasksBuilder.java:371)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:244)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(IncrementalLoadTasksBuilder.java:139)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.executeIncrementalLoad(ReplLoadTask.java:488)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.execute(ReplLoadTask.java:102)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162) 
> 

[jira] [Updated] (HIVE-21679) Replicating a CTAS event creating an MM table fails.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-21679:

Status: Patch Available  (was: Open)

> Replicating a CTAS event creating an MM table fails.
> 
>
> Key: HIVE-21679
> URL: https://issues.apache.org/jira/browse/HIVE-21679
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: CTAS, pull-request-available
> Attachments: HIVE-21679.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> use dumpdb;
> create table t1 (a int, b int);
> insert into t1 values (1, 2), (3, 4);
> create table t6_mm_part partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1
> create table t6_mm stored as orc tblproperties ("transactional"="true", 
> "transactional_properties"="insert_only") as select * from t1;
> repl dump dumpdb;
> create table t6_mm_part_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> create table t6_mm_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> repl dump dumpdb from 
> repl load loaddb from '/tmp/dump/next';
> ERROR : failed replication
> org.apache.hadoop.hive.ql.parse.SemanticException: Invalid table name 
> loaddb.dumpdb.t6_mm_part_2
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2253) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2239) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.plan.AlterTableDesc.setOldName(AlterTableDesc.java:419)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.tableUpdateReplStateTask(IncrementalLoadTasksBuilder.java:286)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.addUpdateReplStateTasks(IncrementalLoadTasksBuilder.java:371)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:244)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(IncrementalLoadTasksBuilder.java:139)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.executeIncrementalLoad(ReplLoadTask.java:488)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.execute(ReplLoadTask.java:102)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:88)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:332)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
>  at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
>  ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
>  at 
> 

[jira] [Updated] (HIVE-21679) Replicating a CTAS event creating an MM table fails.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-21679:

Attachment: HIVE-21679.01.patch

> Replicating a CTAS event creating an MM table fails.
> 
>
> Key: HIVE-21679
> URL: https://issues.apache.org/jira/browse/HIVE-21679
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: CTAS
> Attachments: HIVE-21679.01.patch
>
>
> use dumpdb;
> create table t1 (a int, b int);
> insert into t1 values (1, 2), (3, 4);
> create table t6_mm_part partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1
> create table t6_mm stored as orc tblproperties ("transactional"="true", 
> "transactional_properties"="insert_only") as select * from t1;
> repl dump dumpdb;
> create table t6_mm_part_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> create table t6_mm_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> repl dump dumpdb from 
> repl load loaddb from '/tmp/dump/next';
> ERROR : failed replication
> org.apache.hadoop.hive.ql.parse.SemanticException: Invalid table name 
> loaddb.dumpdb.t6_mm_part_2
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2253) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2239) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.plan.AlterTableDesc.setOldName(AlterTableDesc.java:419)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.tableUpdateReplStateTask(IncrementalLoadTasksBuilder.java:286)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.addUpdateReplStateTasks(IncrementalLoadTasksBuilder.java:371)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:244)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(IncrementalLoadTasksBuilder.java:139)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.executeIncrementalLoad(ReplLoadTask.java:488)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.execute(ReplLoadTask.java:102)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:88)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:332)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
>  at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
>  ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:350)
>  

[jira] [Updated] (HIVE-21679) Replicating a CTAS event creating an MM table fails.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-21679:

Summary: Replicating a CTAS event creating an MM table fails.  (was: 
Replicating a CTAS event creating an MM partitioned table fails)

> Replicating a CTAS event creating an MM table fails.
> 
>
> Key: HIVE-21679
> URL: https://issues.apache.org/jira/browse/HIVE-21679
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: CTAS
>
> use dumpdb;
> create table t1 (a int, b int);
> insert into t1 values (1, 2), (3, 4);
> create table t6_mm_part partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1
> create table t6_mm stored as orc tblproperties ("transactional"="true", 
> "transactional_properties"="insert_only") as select * from t1;
> repl dump dumpdb;
> create table t6_mm_part_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> create table t6_mm_2 partitioned by (a) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only") as select 
> * from t1;
> repl dump dumpdb from 
> repl load loaddb from '/tmp/dump/next';
> ERROR : failed replication
> org.apache.hadoop.hive.ql.parse.SemanticException: Invalid table name 
> loaddb.dumpdb.t6_mm_part_2
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2253) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.Utilities.getDbTableName(Utilities.java:2239) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.plan.AlterTableDesc.setOldName(AlterTableDesc.java:419)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.tableUpdateReplStateTask(IncrementalLoadTasksBuilder.java:286)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.addUpdateReplStateTasks(IncrementalLoadTasksBuilder.java:371)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.analyzeEventLoad(IncrementalLoadTasksBuilder.java:244)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(IncrementalLoadTasksBuilder.java:139)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.executeIncrementalLoad(ReplLoadTask.java:488)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.repl.ReplLoadTask.execute(ReplLoadTask.java:102)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:233)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:88)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at 
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:332)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
>  at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
>  at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
>  ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
>  at 
> 

[jira] [Commented] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834111#comment-16834111
 ] 

Hive QA commented on HIVE-21641:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967925/HIVE-21641.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15981 tests 
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=271)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17136/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17136/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17136/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967925 - PreCommit-HIVE-Build

> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> |   my_avg   |
> ++
> | 37.8923531030581611189434  |
> ++
> {code} 
> Result from Llap external client
> {code}
> +-+
> |   my_avg|
> +-+
> |37.892353|
> +-+
> {code}
>  
> This is due to Driver(beeline path) calls 
> [analyzeInternal()|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L328]
>  for getting result set schema which initializes 
> [resultSchema|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L333]
>  after some more transformations as compared to llap-ext-client which calls 
> [genLogicalPlan()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L561]
> Replacing {{genLogicalPlan()}} by {{analyze()}} resolves this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21680) Backport HIVE-17644 to branch-2 and branch-2.3

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-21680:
--
   Resolution: Fixed
Fix Version/s: 2.3.5
   Status: Resolved  (was: Patch Available)

Ran the tests locally, I didn't see any new failures.  I've committed this to 
branch-2.3.  Note that the title says "Backport to brach 2 and branch 2.3".  
I've only backported this to 2.3.  If it's needed on branch 2 (I'm not sure why 
it would be) a separate Jira should be opened for that.

> Backport HIVE-17644 to branch-2 and branch-2.3
> --
>
> Key: HIVE-21680
> URL: https://issues.apache.org/jira/browse/HIVE-21680
> Project: Hive
>  Issue Type: Bug
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Fix For: 2.3.5
>
> Attachments: HIVE-21680.branch-2.3.patch, HIVE-21680.branch-2.patch
>
>
> Backport HIVE-17644 to fix the warning in {{get statistics when not analyzed 
> in Hive or Spark}}:
> {code:scala}
>   test("get statistics when not analyzed in Hive or Spark") {
> val tabName = "tab1"
> withTable(tabName) {
>   createNonPartitionedTable(tabName, analyzedByHive = false, 
> analyzedBySpark = false)
>   checkTableStats(tabName, hasSizeInBytes = true, expectedRowCounts = 
> None)
>   // ALTER TABLE SET TBLPROPERTIES invalidates some contents of Hive 
> specific statistics
>   // This is triggered by the Hive alterTable API
>   val describeResult = hiveClient.runSqlHive(s"DESCRIBE FORMATTED 
> $tabName")
>   val rawDataSize = extractStatsPropValues(describeResult, "rawDataSize")
>   val numRows = extractStatsPropValues(describeResult, "numRows")
>   val totalSize = extractStatsPropValues(describeResult, "totalSize")
>   assert(rawDataSize.isEmpty, "rawDataSize should not be shown without 
> table analysis")
>   assert(numRows.isEmpty, "numRows should not be shown without table 
> analysis")
>   assert(totalSize.isDefined && totalSize.get > 0, "totalSize is lost")
> }
>   }
> // 
> https://github.com/apache/spark/blob/43dcb91a4cb25aa7e1cc5967194f098029a0361e/sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala#L789-L806
> {code}
> {noformat}
> 06:23:46.103 WARN org.apache.hadoop.hive.metastore.MetaStoreDirectSql: Failed 
> to execute [SELECT "DBS"."NAME", "TBLS"."TBL_NAME", 
> "COLUMNS_V2"."COLUMN_NAME","KEY_CONSTRAINTS"."POSITION", 
> "KEY_CONSTRAINTS"."CONSTRAINT_NAME", "KEY_CONSTRAINTS"."ENABLE_VALIDATE_RELY" 
>  FROM  "TBLS"  INNER  JOIN "KEY_CONSTRAINTS" ON "TBLS"."TBL_ID" = 
> "KEY_CONSTRAINTS"."PARENT_TBL_ID"  INNER JOIN "DBS" ON "TBLS"."DB_ID" = 
> "DBS"."DB_ID"  INNER JOIN "COLUMNS_V2" ON "COLUMNS_V2"."CD_ID" = 
> "KEY_CONSTRAINTS"."PARENT_CD_ID" AND  "COLUMNS_V2"."INTEGER_IDX" = 
> "KEY_CONSTRAINTS"."PARENT_INTEGER_IDX"  WHERE 
> "KEY_CONSTRAINTS"."CONSTRAINT_TYPE" = 0 AND "DBS"."NAME" = ? AND 
> "TBLS"."TBL_NAME" = ?] with parameters [default, tab1]
> javax.jdo.JDODataStoreException: Error executing SQL query "SELECT 
> "DBS"."NAME", "TBLS"."TBL_NAME", 
> "COLUMNS_V2"."COLUMN_NAME","KEY_CONSTRAINTS"."POSITION", 
> "KEY_CONSTRAINTS"."CONSTRAINT_NAME", "KEY_CONSTRAINTS"."ENABLE_VALIDATE_RELY" 
>  FROM  "TBLS"  INNER  JOIN "KEY_CONSTRAINTS" ON "TBLS"."TBL_ID" = 
> "KEY_CONSTRAINTS"."PARENT_TBL_ID"  INNER JOIN "DBS" ON "TBLS"."DB_ID" = 
> "DBS"."DB_ID"  INNER JOIN "COLUMNS_V2" ON "COLUMNS_V2"."CD_ID" = 
> "KEY_CONSTRAINTS"."PARENT_CD_ID" AND  "COLUMNS_V2"."INTEGER_IDX" = 
> "KEY_CONSTRAINTS"."PARENT_INTEGER_IDX"  WHERE 
> "KEY_CONSTRAINTS"."CONSTRAINT_TYPE" = 0 AND "DBS"."NAME" = ? AND 
> "TBLS"."TBL_NAME" = ?".
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
>   at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
>   at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1750)
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPrimaryKeys(MetaStoreDirectSql.java:1939)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:8213)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:8209)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2719)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getPrimaryKeysInternal(ObjectStore.java:8221)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getPrimaryKeys(ObjectStore.java:8199)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> 

[jira] [Updated] (HIVE-21655) Add Re-Try to LdapSearchFactory

[David Mollitor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21655:
--
Description: 
It may be the case that LDAP service is temporarily unreachable.  Please 
implement a re-try facility here:

[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java#L41]

> Add Re-Try to LdapSearchFactory
> ---
>
> Key: HIVE-21655
> URL: https://issues.apache.org/jira/browse/HIVE-21655
> Project: Hive
>  Issue Type: Improvement
>  Components: Authentication
>Affects Versions: 4.0.0, 3.2.0
> Environment: It may be the case that LDAP service is temporarily 
> unreachable.  Please implement a re-try facility here:
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java#L41
>Reporter: David Mollitor
>Priority: Major
>
> It may be the case that LDAP service is temporarily unreachable.  Please 
> implement a re-try facility here:
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java#L41]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write

[David Mollitor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834081#comment-16834081
 ] 

David Mollitor commented on HIVE-21240:
---

[~bslim] Took a few tries, but it did pass tests eventually.  Any more thoughts 
on the matter?

Thanks.

> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, 
> HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.12.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.2.patch, HIVE-21240.3.patch, 
> HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch, 
> HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch, 
> kafka_storage_handler.diff
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats 
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for 
> each row processed, for each column in the row



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13582?focusedWorklogId=237978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237978
 ]

ASF GitHub Bot logged work on HIVE-13582:
-

Author: ASF GitHub Bot
Created on: 06/May/19 18:10
Start Date: 06/May/19 18:10
Worklog Time Spent: 10m 
  Work Description: jcamachor commented on pull request #612: HIVE-13582: 
Support for Quantified predicates for non-correlated queries
URL: https://github.com/apache/hive/pull/612#discussion_r281295724
 
 

 ##
 File path: ql/src/test/results/clientpositive/llap/subquery_ALL.q.out
 ##
 @@ -0,0 +1,542 @@
+PREHOOK: query: create table tempty(i int, j int)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tempty
+POSTHOOK: query: create table tempty(i int, j int)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tempty
+PREHOOK: query: CREATE TABLE part_null_n0 as select * from part
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@part
+PREHOOK: Output: database:default
+PREHOOK: Output: default@part_null_n0
+POSTHOOK: query: CREATE TABLE part_null_n0 as select * from part
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@part
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@part_null_n0
+POSTHOOK: Lineage: part_null_n0.p_brand SIMPLE 
[(part)part.FieldSchema(name:p_brand, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_comment SIMPLE 
[(part)part.FieldSchema(name:p_comment, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_container SIMPLE 
[(part)part.FieldSchema(name:p_container, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_mfgr SIMPLE 
[(part)part.FieldSchema(name:p_mfgr, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_name SIMPLE 
[(part)part.FieldSchema(name:p_name, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_partkey SIMPLE 
[(part)part.FieldSchema(name:p_partkey, type:int, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_retailprice SIMPLE 
[(part)part.FieldSchema(name:p_retailprice, type:double, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_size SIMPLE 
[(part)part.FieldSchema(name:p_size, type:int, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_type SIMPLE 
[(part)part.FieldSchema(name:p_type, type:string, comment:null), ]
+PREHOOK: query: insert into part_null_n0 values(NULL,NULL,NULL,NULL,NULL, 
NULL, NULL,NULL,NULL)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@part_null_n0
+POSTHOOK: query: insert into part_null_n0 values(NULL,NULL,NULL,NULL,NULL, 
NULL, NULL,NULL,NULL)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@part_null_n0
+POSTHOOK: Lineage: part_null_n0.p_brand EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_comment EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_container EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_mfgr EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_name EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_partkey EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_retailprice EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_size EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_type EXPRESSION []
+Warning: Shuffle Join MERGEJOIN[37][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in 
Stage 'Reducer 3' is a cross product
+PREHOOK: query: explain cbo select count(*) from part where p_partkey <> ALL 
(select p_partkey from part)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@part
+ A masked pattern was here 
+POSTHOOK: query: explain cbo select count(*) from part where p_partkey <> ALL 
(select p_partkey from part)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@part
+ A masked pattern was here 
+CBO PLAN:
+HiveAggregate(group=[{}], agg#0=[count()])
+  HiveFilter(condition=[AND(OR(IS NULL($4), =($1, 0)), OR(IS NOT NULL($0), 
=($1, 0), IS NOT NULL($4)), OR(>=($2, $1), =($1, 0), IS NOT NULL($4), IS 
NULL($0)))])
 
 Review comment:
   `HiveFilterJoinRule` should be the rule responsible to push filters into 
join conditions.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 237978)
Time Spent: 1h 40m  (was: 1.5h)

> E061-07 and E061-12: Quantified Comparison Predicates
> -
>
> Key: HIVE-13582
> URL: https://issues.apache.org/jira/browse/HIVE-13582
> Project: Hive
>  Issue Type: Sub-task
>  

[jira] [Commented] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834078#comment-16834078
 ] 

Hive QA commented on HIVE-21641:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
0s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 6 
unchanged - 8 fixed = 6 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17136/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17136/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> | 

[jira] [Updated] (HIVE-21694) Hive driver wait time is fixed for task getting executed in parallel.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-21694:
--
Labels: pull-request-available  (was: )

> Hive driver wait time is fixed for task getting executed in parallel.
> -
>
> Key: HIVE-21694
> URL: https://issues.apache.org/jira/browse/HIVE-21694
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21694.01.patch
>
>
> During a command execution hive driver executes the task in a separate thread 
> if the task to be executed is set as parallel. After starting the task, 
> driver checks if the task has finished execution or not. If the task 
> execution is not finished it waits for 2 seconds before waking up again to 
> check the task status. In case of task with execution time in milliseconds, 
> this wait time can induce substantial overhead. So instead of fixed wait 
> time, exponential backedup sleep time can be used to reduce the sleep 
> overhead. The sleep time can start with 100ms and can increase up to 2 
> seconds doubling on each iteration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21694) Hive driver wait time is fixed for task getting executed in parallel.

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21694:
---
Attachment: HIVE-21694.01.patch

> Hive driver wait time is fixed for task getting executed in parallel.
> -
>
> Key: HIVE-21694
> URL: https://issues.apache.org/jira/browse/HIVE-21694
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21694.01.patch
>
>
> During a command execution hive driver executes the task in a separate thread 
> if the task to be executed is set as parallel. After starting the task, 
> driver checks if the task has finished execution or not. If the task 
> execution is not finished it waits for 2 seconds before waking up again to 
> check the task status. In case of task with execution time in milliseconds, 
> this wait time can induce substantial overhead. So instead of fixed wait 
> time, exponential backedup sleep time can be used to reduce the sleep 
> overhead. The sleep time can start with 100ms and can increase up to 2 
> seconds doubling on each iteration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21694) Hive driver wait time is fixed for task getting executed in parallel.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21694?focusedWorklogId=237966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237966
 ]

ASF GitHub Bot logged work on HIVE-21694:
-

Author: ASF GitHub Bot
Created on: 06/May/19 17:42
Start Date: 06/May/19 17:42
Worklog Time Spent: 10m 
  Work Description: maheshk114 commented on pull request #616: HIVE-21694 : 
Hive driver wait time is fixed for task getting executed in parallel.
URL: https://github.com/apache/hive/pull/616
 
 
   … 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 237966)
Time Spent: 10m
Remaining Estimate: 0h

> Hive driver wait time is fixed for task getting executed in parallel.
> -
>
> Key: HIVE-21694
> URL: https://issues.apache.org/jira/browse/HIVE-21694
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21694.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> During a command execution hive driver executes the task in a separate thread 
> if the task to be executed is set as parallel. After starting the task, 
> driver checks if the task has finished execution or not. If the task 
> execution is not finished it waits for 2 seconds before waking up again to 
> check the task status. In case of task with execution time in milliseconds, 
> this wait time can induce substantial overhead. So instead of fixed wait 
> time, exponential backedup sleep time can be used to reduce the sleep 
> overhead. The sleep time can start with 100ms and can increase up to 2 
> seconds doubling on each iteration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21694) Hive driver wait time is fixed for task getting executed in parallel.

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21694:
---
Status: Patch Available  (was: Open)

> Hive driver wait time is fixed for task getting executed in parallel.
> -
>
> Key: HIVE-21694
> URL: https://issues.apache.org/jira/browse/HIVE-21694
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21694.01.patch
>
>
> During a command execution hive driver executes the task in a separate thread 
> if the task to be executed is set as parallel. After starting the task, 
> driver checks if the task has finished execution or not. If the task 
> execution is not finished it waits for 2 seconds before waking up again to 
> check the task status. In case of task with execution time in milliseconds, 
> this wait time can induce substantial overhead. So instead of fixed wait 
> time, exponential backedup sleep time can be used to reduce the sleep 
> overhead. The sleep time can start with 100ms and can increase up to 2 
> seconds doubling on each iteration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834034#comment-16834034
 ] 

Hive QA commented on HIVE-21240:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967923/HIVE-21240.12.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15990 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17135/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17135/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17135/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967923 - PreCommit-HIVE-Build

> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, 
> HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.12.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.2.patch, HIVE-21240.3.patch, 
> HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch, 
> HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch, 
> kafka_storage_handler.diff
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats 
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for 
> each row processed, for each column in the row



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-21344:
---
Description: 
For every query, {{getAllValidMaterializedViews}} still requires a call to 
metastore to verify that the materializations exist, whether they are outdated 
or not, etc. Since this is only useful for active-active HS2 deployments, we 
could take a less aggressive approach and check this information only after 
rewriting has been triggered. In addition, we could refresh the information in 
the HS2 registry periodically in a background thread.

{code}
// This is not a rebuild, we retrieve all the materializations. In turn, we do 
not need
// to force the materialization contents to be up-to-date, as this is not a 
rebuild, and
// we apply the user parameters (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) 
instead.
materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan), 
false, getTxnMgr());
{code}

!mv-get-from-remote.png!

  was:
registry is not used for Calcite planner.

{code}
// This is not a rebuild, we retrieve all the materializations. In turn, we do 
not need
  // to force the materialization contents to be up-to-date, as this is 
not a rebuild, and
  // we apply the user parameters 
(HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
  materializations = 
db.getAllValidMaterializedViews(getTablesUsed(basePlan), false, getTxnMgr());
}
{code}

!mv-get-from-remote.png!


> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to 
> metastore to verify that the materializations exist, whether they are 
> outdated or not, etc. Since this is only useful for active-active HS2 
> deployments, we could take a less aggressive approach and check this 
> information only after rewriting has been triggered. In addition, we could 
> refresh the information in the HS2 registry periodically in a background 
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we 
> do not need
> // to force the materialization contents to be up-to-date, as this is not a 
> rebuild, and
> // we apply the user parameters 
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan), 
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-21344:
---
Summary: CBO: Reduce compilation time in presence of materialized views  
(was: CBO: Materialized view registry is not used for Calcite planner)

> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we 
> do not need
>   // to force the materialization contents to be up-to-date, as this 
> is not a rebuild, and
>   // we apply the user parameters 
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
>   materializations = 
> db.getAllValidMaterializedViews(getTablesUsed(basePlan), false, getTxnMgr());
> }
> {code}
> !mv-get-from-remote.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-21344:
---
Description: 
registry is not used for Calcite planner.

{code}
// This is not a rebuild, we retrieve all the materializations. In turn, we do 
not need
  // to force the materialization contents to be up-to-date, as this is 
not a rebuild, and
  // we apply the user parameters 
(HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
  materializations = 
db.getAllValidMaterializedViews(getTablesUsed(basePlan), false, getTxnMgr());
}
{code}

!mv-get-from-remote.png!

  was:
{code}
// This is not a rebuild, we retrieve all the materializations. In turn, we do 
not need
  // to force the materialization contents to be up-to-date, as this is 
not a rebuild, and
  // we apply the user parameters 
(HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
  materializations = 
db.getAllValidMaterializedViews(getTablesUsed(basePlan), false, getTxnMgr());
}
{code}

!mv-get-from-remote.png!


> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
>  Issue Type: Bug
>  Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>
> registry is not used for Calcite planner.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we 
> do not need
>   // to force the materialization contents to be up-to-date, as this 
> is not a rebuild, and
>   // we apply the user parameters 
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
>   materializations = 
> db.getAllValidMaterializedViews(getTablesUsed(basePlan), false, getTxnMgr());
> }
> {code}
> !mv-get-from-remote.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21685) Wrong simplification in query with multiple IN clauses

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-21685:
---
Attachment: HIVE-21685.patch

> Wrong simplification in query with multiple IN clauses
> --
>
> Key: HIVE-21685
> URL: https://issues.apache.org/jira/browse/HIVE-21685
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Oliver Draese
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21685.patch, HIVE-21685.patch, HIVE-21685.patch, 
> HIVE-21685.patch
>
>
> Simple test to reproduce:
> {code}
> select * from table1 where name IN(‘g’,‘r’) AND name IN(‘a’,‘b’);
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21625) Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21625:
--
Attachment: HIVE-21625.4.patch

> Fix TxnIdUtils.checkEquivalentWriteIds, also provides a comparison method
> -
>
> Key: HIVE-21625
> URL: https://issues.apache.org/jira/browse/HIVE-21625
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21625.1.patch, HIVE-21625.2.patch, 
> HIVE-21625.3.patch, HIVE-21625.4.patch
>
>
> TxnIdUtils.checkEquivalentWriteIds has a bug which thinks (\{1,2,3,4\}, 6) 
> and (\{1,2,3,4,5,6\}, 8) compatible (the notation is (invalidlist, hwm)). 
> Here is a patch to fix it, also provide a comparison method to check which is 
> newer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833995#comment-16833995
 ] 

Hive QA commented on HIVE-21240:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
40s{color} | {color:blue} serde in master has 197 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
57s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} hcatalog/core in master has 28 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} serde: The patch generated 0 new + 4 unchanged - 25 
fixed = 4 total (was 29) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} ql: The patch generated 0 new + 6 unchanged - 5 
fixed = 6 total (was 11) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
49s{color} | {color:green} serde generated 0 new + 193 unchanged - 4 fixed = 
193 total (was 197) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
4s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m  3s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17135/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde ql hcatalog/core U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17135/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: David Mollitor
> 

[jira] [Updated] (HIVE-21696) Include column stats in explain cbo formatted

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-21696:
---
Status: Patch Available  (was: In Progress)

> Include column stats in explain cbo formatted
> -
>
> Key: HIVE-21696
> URL: https://issues.apache.org/jira/browse/HIVE-21696
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21696.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21696) Include column stats in explain cbo formatted

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-21696:
---
Attachment: HIVE-21696.patch

> Include column stats in explain cbo formatted
> -
>
> Key: HIVE-21696
> URL: https://issues.apache.org/jira/browse/HIVE-21696
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21696.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-21696) Include column stats in explain cbo formatted

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-21696:
--


> Include column stats in explain cbo formatted
> -
>
> Key: HIVE-21696
> URL: https://issues.apache.org/jira/browse/HIVE-21696
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-21696) Include column stats in explain cbo formatted

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21696 started by Jesus Camacho Rodriguez.
--
> Include column stats in explain cbo formatted
> -
>
> Key: HIVE-21696
> URL: https://issues.apache.org/jira/browse/HIVE-21696
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21671) Replicate Streaming ingestion with transactional batch size as 1.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833960#comment-16833960
 ] 

Hive QA commented on HIVE-21671:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967922/HIVE-21671.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15983 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17134/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17134/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17134/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967922 - PreCommit-HIVE-Build

> Replicate Streaming ingestion with transactional batch size as 1.
> -
>
> Key: HIVE-21671
> URL: https://issues.apache.org/jira/browse/HIVE-21671
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Streaming, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Attachments: HIVE-21671.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Replication streaming ingest HiveStreamingConnection on ACID tables with 
> transaction batch size as 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21693) Break up DDLTask - extract Process related operations

[Zoltan Haindrich (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833951#comment-16833951
 ] 

Zoltan Haindrich commented on HIVE-21693:
-

do we need to keep the useNewFormat in  showLocks? Can't we use the "new" 
format only and forget the old?
+1

> Break up DDLTask - extract Process related operations
> -
>
> Key: HIVE-21693
> URL: https://issues.apache.org/jira/browse/HIVE-21693
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>  Labels: refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21693.01.patch, HIVE-21693.02.patch, 
> HIVE-21693.03.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is 
> also a huge class, which has a field for each DDL operation it supports. The 
> goal is to refactor these in order to have everything cut into more 
> handleable classes under the package  org.apache.hadoop.hive.ql.exec.ddl:
>  * have a separate class for each operation
>  * have a package for each operation group (database ddl, table ddl, etc), so 
> the amount of classes under a package is more manageable
>  * make all the requests (DDLDesc subclasses) immutable
>  * DDLTask should be agnostic to the actual operations
>  * right now let's ignore the issue of having some operations handled by 
> DDLTask which are not actual DDL operations (lock, unlock, desc...)
> In the interim time when there are two DDLTask and DDLWork classes in the 
> code base the new ones in the new package are called DDLTask2 and DDLWork2 
> thus avoiding the usage of fully qualified class names where both the old and 
> the new classes are in use.
> Step #7: extract all the process related operations from the old DDLTask, and 
> move them under the new package.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21671) Replicate Streaming ingestion with transactional batch size as 1.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833920#comment-16833920
 ] 

Hive QA commented on HIVE-21671:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
5s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} streaming: The patch generated 2 new + 31 unchanged - 
1 fixed = 33 total (was 32) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} itests/hive-unit: The patch generated 28 new + 0 
unchanged - 0 fixed = 28 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
37s{color} | {color:red} streaming generated 1 new + 2 unchanged - 0 fixed = 3 
total (was 2) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 24s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:streaming |
|  |  Should org.apache.hive.streaming.HiveStreamingConnection$WriteDirInfo be 
a _static_ inner class?  At HiveStreamingConnection.java:inner class?  At 
HiveStreamingConnection.java:[lines 700-710] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17134/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17134/yetus/diff-checkstyle-streaming.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17134/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17134/yetus/new-findbugs-streaming.html
 |
| modules | C: ql streaming itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17134/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Replicate Streaming ingestion with transactional batch size as 1.
> -
>
> Key: HIVE-21671
>   

[jira] [Commented] (HIVE-21662) Refactor some functionality from QTestUtil to separate classes

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833886#comment-16833886
 ] 

Hive QA commented on HIVE-21662:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967921/HIVE-21662.04.patch

{color:green}SUCCESS:{color} +1 due to 16 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15980 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
 (batchId=164)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_16]
 (batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket4] 
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7] 
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[disable_merge_for_bucketing]
 (batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[leftsemijoin_mr] 
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[scriptfile1] 
(batchId=150)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17133/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17133/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17133/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967921 - PreCommit-HIVE-Build

> Refactor some functionality from QTestUtil to separate classes
> --
>
> Key: HIVE-21662
> URL: https://issues.apache.org/jira/browse/HIVE-21662
> Project: Hive
>  Issue Type: Improvement
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21662.01.patch, HIVE-21662.02.patch, 
> HIVE-21662.03.patch, HIVE-21662.04.patch, HIVE-21662.04.patch
>
>
> This may be a follow-up of HIVE-18560:
> new classes: QTestSystemProperties, QFileVersionHandler, QTestMiniClusters, 
> QTestDatasetHandler, QTestResultProcessor



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21152) Rewrite if expression to case and recognize simple case as an if

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-21152:

Attachment: HIVE-21152.07.patch

> Rewrite if expression to case and recognize simple case as an if
> 
>
> Key: HIVE-21152
> URL: https://issues.apache.org/jira/browse/HIVE-21152
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21152.01.patch, HIVE-21152.02.patch, 
> HIVE-21152.03.patch, HIVE-21152.04.patch, HIVE-21152.05.patch, 
> HIVE-21152.06.patch, HIVE-21152.07.patch
>
>
> * {{IF}} is not part of the sql standard; however given its special form its 
> simpler - and currently in Hive it also has vectorized support
> * people writing standard sql may write: {{CASE WHEN member=1 THEN attr+1 
> else attr+2 END}} which is essentially an if.
> The idea is to rewrite IFs to CASEs for the cbo; and recognize simple 
> "CASE"-s as IFs to get vectorization on them if possible



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-21304:

Attachment: HIVE-21304.05.patch

> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, 
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19353) Vectorization: ConstantVectorExpression --> RuntimeException: Unexpected column vector type LIST

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-19353:

Attachment: HIVE-19353.04.patch

> Vectorization: ConstantVectorExpression  --> RuntimeException: Unexpected 
> column vector type LIST
> -
>
> Key: HIVE-19353
> URL: https://issues.apache.org/jira/browse/HIVE-19353
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19353.01.patch, HIVE-19353.02.patch, 
> HIVE-19353.03.patch, HIVE-19353.04.patch
>
>
> Found by enabling vectorization for 
> org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
> {noformat}
> Caused by: java.lang.RuntimeException: Unexpected column vector type LIST
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression.evaluate(ConstantVectorExpression.java:237)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.closeOp(VectorMapOperator.java:984)
>  ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) 
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21662) Refactor some functionality from QTestUtil to separate classes

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833849#comment-16833849
 ] 

Hive QA commented on HIVE-21662:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
43s{color} | {color:blue} itests/util in master has 46 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} itests/util: The patch generated 33 new + 194 
unchanged - 43 fixed = 227 total (was 237) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
52s{color} | {color:red} itests/util generated 2 new + 42 unchanged - 4 fixed = 
44 total (was 46) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 28s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:itests/util |
|  |  Exceptional return value of java.io.File.delete() ignored in 
org.apache.hadoop.hive.ql.QTestResultProcessor.executeDiffCommand(String, 
String, boolean, String)  At QTestResultProcessor.java:ignored in 
org.apache.hadoop.hive.ql.QTestResultProcessor.executeDiffCommand(String, 
String, boolean, String)  At QTestResultProcessor.java:[line 141] |
|  |  org.apache.hadoop.hive.ql.QTestUtil.DEBUG_HINT isn't final but should be  
At QTestUtil.java:be  At QTestUtil.java:[line 94] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17133/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17133/yetus/diff-checkstyle-itests_util.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17133/yetus/whitespace-eol.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17133/yetus/new-findbugs-itests_util.html
 |
| modules | C: itests/util itests/hive-unit U: itests |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17133/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Refactor some functionality from QTestUtil to separate classes
> --
>
> Key: 

[jira] [Updated] (HIVE-21576) Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats

[Karen Coppage (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karen Coppage updated HIVE-21576:
-
Attachment: HIVE-21576.2.patch
Status: Patch Available  (was: Open)

> Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats
> -
>
> Key: HIVE-21576
> URL: https://issues.apache.org/jira/browse/HIVE-21576
> Project: Hive
>  Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21576.01.patch, HIVE-21576.2.patch
>
>
> Introduce FORMAT clause to CAST statements as well as the below limited list 
> of SQL:2016 datetime formats to Hive in general. These can be used if a 
> session-level feature flag is turned on.
>  * 
>  * MM
>  * DD
>  * HH
>  * MI
>  * SS
> Definitions of these formats here: 
> [https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20967) Handle alter events when replicate to cluster with hive.strict.managed.tables enabled.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833830#comment-16833830
 ] 

Hive QA commented on HIVE-20967:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967919/HIVE-20967.04.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15983 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
 (batchId=247)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17132/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17132/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17132/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967919 - PreCommit-HIVE-Build

> Handle alter events when replicate to cluster with hive.strict.managed.tables 
> enabled.
> --
>
> Key: HIVE-20967
> URL: https://issues.apache.org/jira/browse/HIVE-20967
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: Ashutosh Bapat
>Priority: Minor
>  Labels: DR, pull-request-available
> Attachments: HIVE-20967.01.patch, HIVE-20967.03.patch, 
> HIVE-20967.03.patch, HIVE-20967.04.patch, HIVE-21678.02.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Some of the events from Hive2 may cause conflicts in Hive3 
> (hive.strict.managed.tables=true) when applied. So, need to handle them 
> properly.
>  1. Alter table to convert non-acid to acid.
>  - Do not allow this conversion on source of replication if strict.managed is 
> false.
> 2. Alter table or partition that changes the location.
>  - For managed tables at source, the table location shouldn't be changed for 
> the given non-partitioned table and partition location shouldn't be changed 
> for partitioned table as alter event doesn't capture the new files list. So, 
> it may cause data inconsistsency. So, if database is enabled for replication 
> at source, then alter location on managed tables should be blocked.
>  - For external partitioned tables, if location is changed at source, the the 
> location should be changed for the table and any partitions which reside 
> within the table location, but not for the partitions which are not within 
> the table location. (may be we just need the test).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20967) Handle alter events when replicate to cluster with hive.strict.managed.tables enabled.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833802#comment-16833802
 ] 

Hive QA commented on HIVE-20967:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
33s{color} | {color:blue} standalone-metastore/metastore-common in master has 
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m  
6s{color} | {color:blue} standalone-metastore/metastore-server in master has 
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
52s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} standalone-metastore/metastore-server: The patch 
generated 5 new + 26 unchanged - 0 fixed = 31 total (was 26) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 1 new + 59 unchanged - 1 fixed 
= 60 total (was 60) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  8m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 15s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17132/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17132/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17132/yetus/diff-checkstyle-ql.txt
 |
| modules | C: standalone-metastore/metastore-common 
standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17132/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Handle alter events when replicate to cluster with hive.strict.managed.tables 
> enabled.
> --
>
> Key: HIVE-20967
> URL: 

[jira] [Commented] (HIVE-21576) Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats

[Shawn Weeks (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833785#comment-16833785
 ] 

Shawn Weeks commented on HIVE-21576:


Is there any plan to include fractional seconds. That seems to be consistently 
missing across all the time stamp functions. 

> Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats
> -
>
> Key: HIVE-21576
> URL: https://issues.apache.org/jira/browse/HIVE-21576
> Project: Hive
>  Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21576.01.patch
>
>
> Introduce FORMAT clause to CAST statements as well as the below limited list 
> of SQL:2016 datetime formats to Hive in general. These can be used if a 
> session-level feature flag is turned on.
>  * 
>  * MM
>  * DD
>  * HH
>  * MI
>  * SS
> Definitions of these formats here: 
> [https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21641) Llap external client returns decimal columns in different precision/scale as compared to beeline

[Shubham Chaurasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shubham Chaurasia updated HIVE-21641:
-
Attachment: HIVE-21641.3.patch

> Llap external client returns decimal columns in different precision/scale as 
> compared to beeline
> 
>
> Key: HIVE-21641
> URL: https://issues.apache.org/jira/browse/HIVE-21641
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21641.1.patch, HIVE-21641.2.patch, 
> HIVE-21641.3.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Llap external client gives different precision/scale as compared to when the 
> query is executed beeline. Consider the following results:
> Query:
> {code} 
> select avg(ss_ext_sales_price) my_avg from store_sales;
> {code} 
> Result from Beeline
> {code} 
> ++
> |   my_avg   |
> ++
> | 37.8923531030581611189434  |
> ++
> {code} 
> Result from Llap external client
> {code}
> +-+
> |   my_avg|
> +-+
> |37.892353|
> +-+
> {code}
>  
> This is due to Driver(beeline path) calls 
> [analyzeInternal()|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L328]
>  for getting result set schema which initializes 
> [resultSchema|https://github.com/apache/hive/blob/rel/release-3.1.1/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L333]
>  after some more transformations as compared to llap-ext-client which calls 
> [genLogicalPlan()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L561]
> Replacing {{genLogicalPlan()}} by {{analyze()}} resolves this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20967) Handle alter events when replicate to cluster with hive.strict.managed.tables enabled.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833748#comment-16833748
 ] 

Hive QA commented on HIVE-20967:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967915/HIVE-20967.03.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 15953 tests 
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=179)

[vector_interval_2.q,schema_evol_orc_acid_part_update.q,orc_ppd_varchar.q,strict_managed_tables2.q,vectorization_div0.q,limit_pushdown.q,auto_join_nulls.q,metadata_only_queries_with_filters.q,subquery_notin.q,schema_evol_text_nonvec_part_all_complex.q,orc_schema_evol_quoted.q,vector_between_columns.q,results_cache_with_masking.q,vector_groupby_grouping_sets6.q,bucket_num_reducers.q,materialized_view_rewrite_6.q,update_after_multiple_inserts.q,offset_limit_ppd_optimizer.q,mapjoin_decimal.q,materialized_view_describe.q,orc_merge_incompat1.q,explainanalyze_2.q,schema_evol_text_vecrow_part_all_complex_llap_io.q,external_jdbc_table4.q,vector_char_2.q,column_name_is_table_alias.q,acid_no_buckets.q,extrapolate_part_stats_partial_ndv.q,auto_sortmerge_join_9.q,vector_groupby_grouping_id2.q]
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.retryIncBootstrapAcidFromDifferentDumpWithoutCleanTablesConfig
 (batchId=247)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncremental
 (batchId=247)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testAcidTablesBootstrapDuringIncrementalWithOpenTxnsTimeout
 (batchId=247)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
 (batchId=247)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testRetryAcidTablesBootstrapFromDifferentDump
 (batchId=247)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesBootstrap
 (batchId=258)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesMoveOptimizationBootStrap
 (batchId=258)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testAcidTablesMoveOptimizationIncremental
 (batchId=258)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesBootstrap
 (batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesMoveOptimizationBootStrap
 (batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testAcidTablesMoveOptimizationIncremental
 (batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.retryIncBootstrapAcidFromDifferentDumpWithoutCleanTablesConfig
 (batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testAcidTablesBootstrapDuringIncremental
 (batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testAcidTablesBootstrapDuringIncrementalWithOpenTxnsTimeout
 (batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites
 (batchId=245)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testRetryAcidTablesBootstrapFromDifferentDump
 (batchId=245)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/17131/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17131/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17131/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12967915 - PreCommit-HIVE-Build

> Handle alter events when replicate to cluster with hive.strict.managed.tables 
> enabled.
> --
>
> Key: HIVE-20967
> URL: https://issues.apache.org/jira/browse/HIVE-20967
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: Ashutosh Bapat
>Priority: Minor
>  Labels: DR, pull-request-available
> Attachments: HIVE-20967.01.patch, HIVE-20967.03.patch, 
> HIVE-20967.03.patch, HIVE-20967.04.patch, HIVE-21678.02.patch
>
> 

[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write

[David Mollitor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21240:
--
Attachment: HIVE-21240.12.patch

> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, 
> HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.12.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.2.patch, HIVE-21240.3.patch, 
> HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch, 
> HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch, 
> kafka_storage_handler.diff
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats 
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for 
> each row processed, for each column in the row



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write

[David Mollitor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21240:
--
Status: Patch Available  (was: Open)

> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 3.1.1, 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, 
> HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.12.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.2.patch, HIVE-21240.3.patch, 
> HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch, 
> HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch, 
> kafka_storage_handler.diff
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats 
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for 
> each row processed, for each column in the row



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write

[David Mollitor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21240:
--
Status: Open  (was: Patch Available)

> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 3.1.1, 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, 
> HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.12.patch, 
> HIVE-21240.12.patch, HIVE-21240.12.patch, HIVE-21240.12.patch, 
> HIVE-21240.2.patch, HIVE-21240.3.patch, HIVE-21240.4.patch, 
> HIVE-21240.5.patch, HIVE-21240.6.patch, HIVE-21240.7.patch, 
> HIVE-21240.9.patch, HIVE-24240.8.patch, kafka_storage_handler.diff
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats 
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for 
> each row processed, for each column in the row



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21678) CTAS creating a partitioned table fails because of no writeId

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21678?focusedWorklogId=237736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237736
 ]

ASF GitHub Bot logged work on HIVE-21678:
-

Author: ASF GitHub Bot
Created on: 06/May/19 11:49
Start Date: 06/May/19 11:49
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #614: HIVE-21678
URL: https://github.com/apache/hive/pull/614#discussion_r281151528
 
 

 ##
 File path: ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
 ##
 @@ -398,11 +398,20 @@ public int execute(DriverContext driverContext) {
 // for transactional table if write id is not set during replication 
from a cluster with STRICT_MANAGED set
 // to false then set it now.
 if (tbd.getWriteId() <= 0 && 
AcidUtils.isTransactionalTable(table.getParameters())) {
-  Long writeId = ReplUtils.getMigrationCurrentTblWriteId(conf);
-  if (writeId == null) {
-throw new HiveException("MoveTask : Write id is not set in the 
config by open txn task for migration");
+  if (work.getIsInReplicationScope()) {
+Long writeId = ReplUtils.getMigrationCurrentTblWriteId(conf);
+if (writeId == null) {
+  throw new HiveException("MoveTask : Write id is not set in the 
config by open txn task for migration");
+}
+tbd.setWriteId(writeId);
+  } else if (tbd.isCtas()) {
+// In CTAS flow, the table may not have got created when creating 
load table task.
+// So, fetch the writeId here from the table.
+tbd.setWriteId(table.getTTable().getWriteId());
 
 Review comment:
   Shall use 
driverContext.getCtx().getHiveTxnManager().getAllocatedTableWriteId as we are 
not sure if writeId is always set in table object. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 237736)
Time Spent: 50m  (was: 40m)

> CTAS creating a partitioned table fails because of no writeId
> -
>
> Key: HIVE-21678
> URL: https://issues.apache.org/jira/browse/HIVE-21678
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21678.01.patch, HIVE-21678.02.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> create table t1(a int, b int);
> insert into t1 values (1, 2), (3, 4);
> create table t6_part partitioned by (a) stored as orc tblproperties 
> ("transactional"="true") as select * from t1;
> ERROR : FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.MoveTask. MoveTask : Write id is not set in 
> the config by open txn task for migration
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. MoveTask : Write id is not 
> set in the config by open txn task for migration (state=08S01,code=1)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21678) CTAS creating a partitioned table fails because of no writeId

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21678?focusedWorklogId=237735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237735
 ]

ASF GitHub Bot logged work on HIVE-21678:
-

Author: ASF GitHub Bot
Created on: 06/May/19 11:49
Start Date: 06/May/19 11:49
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #614: HIVE-21678
URL: https://github.com/apache/hive/pull/614#discussion_r281151762
 
 

 ##
 File path: ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
 ##
 @@ -398,11 +398,20 @@ public int execute(DriverContext driverContext) {
 // for transactional table if write id is not set during replication 
from a cluster with STRICT_MANAGED set
 // to false then set it now.
 if (tbd.getWriteId() <= 0 && 
AcidUtils.isTransactionalTable(table.getParameters())) {
-  Long writeId = ReplUtils.getMigrationCurrentTblWriteId(conf);
-  if (writeId == null) {
-throw new HiveException("MoveTask : Write id is not set in the 
config by open txn task for migration");
+  if (work.getIsInReplicationScope()) {
+Long writeId = ReplUtils.getMigrationCurrentTblWriteId(conf);
+if (writeId == null) {
+  throw new HiveException("MoveTask : Write id is not set in the 
config by open txn task for migration");
+}
+tbd.setWriteId(writeId);
+  } else if (tbd.isCtas()) {
+// In CTAS flow, the table may not have got created when creating 
load table task.
+// So, fetch the writeId here from the table.
+tbd.setWriteId(table.getTTable().getWriteId());
+  } else {
+throw new HiveException("MoveTask : Write id is not provided for 
transactiona table " +
 
 Review comment:
   Spelling error in "transactiona".
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 237735)
Time Spent: 40m  (was: 0.5h)

> CTAS creating a partitioned table fails because of no writeId
> -
>
> Key: HIVE-21678
> URL: https://issues.apache.org/jira/browse/HIVE-21678
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21678.01.patch, HIVE-21678.02.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> create table t1(a int, b int);
> insert into t1 values (1, 2), (3, 4);
> create table t6_part partitioned by (a) stored as orc tblproperties 
> ("transactional"="true") as select * from t1;
> ERROR : FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.MoveTask. MoveTask : Write id is not set in 
> the config by open txn task for migration
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. MoveTask : Write id is not 
> set in the config by open txn task for migration (state=08S01,code=1)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21678) CTAS creating a partitioned table fails because of no writeId

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21678?focusedWorklogId=237734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237734
 ]

ASF GitHub Bot logged work on HIVE-21678:
-

Author: ASF GitHub Bot
Created on: 06/May/19 11:49
Start Date: 06/May/19 11:49
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #614: HIVE-21678
URL: https://github.com/apache/hive/pull/614#discussion_r281152390
 
 

 ##
 File path: ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
 ##
 @@ -398,11 +398,20 @@ public int execute(DriverContext driverContext) {
 // for transactional table if write id is not set during replication 
from a cluster with STRICT_MANAGED set
 // to false then set it now.
 if (tbd.getWriteId() <= 0 && 
AcidUtils.isTransactionalTable(table.getParameters())) {
-  Long writeId = ReplUtils.getMigrationCurrentTblWriteId(conf);
-  if (writeId == null) {
-throw new HiveException("MoveTask : Write id is not set in the 
config by open txn task for migration");
+  if (work.getIsInReplicationScope()) {
+Long writeId = ReplUtils.getMigrationCurrentTblWriteId(conf);
+if (writeId == null) {
+  throw new HiveException("MoveTask : Write id is not set in the 
config by open txn task for migration");
+}
+tbd.setWriteId(writeId);
+  } else if (tbd.isCtas()) {
 
 Review comment:
   Need not pass isCtas flag. Instead 
driverContext.getCtx().getHiveTxnManager().getAllocatedTableWriteId can be 
called and if it returns 0, then throw error.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 237734)
Time Spent: 40m  (was: 0.5h)

> CTAS creating a partitioned table fails because of no writeId
> -
>
> Key: HIVE-21678
> URL: https://issues.apache.org/jira/browse/HIVE-21678
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21678.01.patch, HIVE-21678.02.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> create table t1(a int, b int);
> insert into t1 values (1, 2), (3, 4);
> create table t6_part partitioned by (a) stored as orc tblproperties 
> ("transactional"="true") as select * from t1;
> ERROR : FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.MoveTask. MoveTask : Write id is not set in 
> the config by open txn task for migration
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. MoveTask : Write id is not 
> set in the config by open txn task for migration (state=08S01,code=1)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20967) Handle alter events when replicate to cluster with hive.strict.managed.tables enabled.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833728#comment-16833728
 ] 

Hive QA commented on HIVE-20967:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
27s{color} | {color:blue} standalone-metastore/metastore-common in master has 
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m 
12s{color} | {color:blue} standalone-metastore/metastore-server in master has 
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
59s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
18s{color} | {color:red} standalone-metastore/metastore-server: The patch 
generated 5 new + 26 unchanged - 0 fixed = 31 total (was 26) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 1 new + 59 unchanged - 1 fixed 
= 60 total (was 60) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  8m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17131/dev-support/hive-personality.sh
 |
| git revision | master / c5e6608 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17131/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17131/yetus/diff-checkstyle-ql.txt
 |
| modules | C: standalone-metastore/metastore-common 
standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-17131/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Handle alter events when replicate to cluster with hive.strict.managed.tables 
> enabled.
> --
>
> Key: HIVE-20967
> URL: 

[jira] [Updated] (HIVE-21671) Replicate Streaming ingestion with transactional batch size as 1.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-21671:

Status: Patch Available  (was: Open)

> Replicate Streaming ingestion with transactional batch size as 1.
> -
>
> Key: HIVE-21671
> URL: https://issues.apache.org/jira/browse/HIVE-21671
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Streaming, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Attachments: HIVE-21671.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Replication streaming ingest HiveStreamingConnection on ACID tables with 
> transaction batch size as 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (HIVE-21671) Replicate Streaming ingestion with transactional batch size as 1.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21671?focusedWorklogId=237730=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237730
 ]

ASF GitHub Bot logged work on HIVE-21671:
-

Author: ASF GitHub Bot
Created on: 06/May/19 11:27
Start Date: 06/May/19 11:27
Worklog Time Spent: 10m 
  Work Description: sankarh commented on pull request #615: HIVE-21671: 
Replicate Streaming ingestion with transactional batch size as 1.
URL: https://github.com/apache/hive/pull/615
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 237730)
Time Spent: 10m
Remaining Estimate: 0h

> Replicate Streaming ingestion with transactional batch size as 1.
> -
>
> Key: HIVE-21671
> URL: https://issues.apache.org/jira/browse/HIVE-21671
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Streaming, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Attachments: HIVE-21671.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Replication streaming ingest HiveStreamingConnection on ACID tables with 
> transaction batch size as 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21671) Replicate Streaming ingestion with transactional batch size as 1.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-21671:

Attachment: HIVE-21671.01.patch

> Replicate Streaming ingestion with transactional batch size as 1.
> -
>
> Key: HIVE-21671
> URL: https://issues.apache.org/jira/browse/HIVE-21671
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Streaming, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Attachments: HIVE-21671.01.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Replication streaming ingest HiveStreamingConnection on ACID tables with 
> transaction batch size as 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21671) Replicate Streaming ingestion with transactional batch size as 1.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-21671:
--
Labels: DR pull-request-available replication  (was: DR replication)

> Replicate Streaming ingestion with transactional batch size as 1.
> -
>
> Key: HIVE-21671
> URL: https://issues.apache.org/jira/browse/HIVE-21671
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Streaming, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: DR, pull-request-available, replication
> Attachments: HIVE-21671.01.patch
>
>
> Replication streaming ingest HiveStreamingConnection on ACID tables with 
> transaction batch size as 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-21662) Refactor some functionality from QTestUtil to separate classes

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-21662:

Attachment: HIVE-21662.04.patch

> Refactor some functionality from QTestUtil to separate classes
> --
>
> Key: HIVE-21662
> URL: https://issues.apache.org/jira/browse/HIVE-21662
> Project: Hive
>  Issue Type: Improvement
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21662.01.patch, HIVE-21662.02.patch, 
> HIVE-21662.03.patch, HIVE-21662.04.patch, HIVE-21662.04.patch
>
>
> This may be a follow-up of HIVE-18560:
> new classes: QTestSystemProperties, QFileVersionHandler, QTestMiniClusters, 
> QTestDatasetHandler, QTestResultProcessor



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21662) Refactor some functionality from QTestUtil to separate classes

[Laszlo Bodor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833714#comment-16833714
 ] 

Laszlo Bodor commented on HIVE-21662:
-

TestSparkCliDriver tests pass locally, reattaching 04.patch

> Refactor some functionality from QTestUtil to separate classes
> --
>
> Key: HIVE-21662
> URL: https://issues.apache.org/jira/browse/HIVE-21662
> Project: Hive
>  Issue Type: Improvement
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21662.01.patch, HIVE-21662.02.patch, 
> HIVE-21662.03.patch, HIVE-21662.04.patch, HIVE-21662.04.patch
>
>
> This may be a follow-up of HIVE-18560:
> new classes: QTestSystemProperties, QFileVersionHandler, QTestMiniClusters, 
> QTestDatasetHandler, QTestResultProcessor



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20967) Handle alter events when replicate to cluster with hive.strict.managed.tables enabled.

[Ashutosh Bapat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-20967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Bapat updated HIVE-20967:
--
Attachment: HIVE-20967.04.patch
Status: Patch Available  (was: In Progress)

> Handle alter events when replicate to cluster with hive.strict.managed.tables 
> enabled.
> --
>
> Key: HIVE-20967
> URL: https://issues.apache.org/jira/browse/HIVE-20967
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: Ashutosh Bapat
>Priority: Minor
>  Labels: DR, pull-request-available
> Attachments: HIVE-20967.01.patch, HIVE-20967.03.patch, 
> HIVE-20967.03.patch, HIVE-20967.04.patch, HIVE-21678.02.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Some of the events from Hive2 may cause conflicts in Hive3 
> (hive.strict.managed.tables=true) when applied. So, need to handle them 
> properly.
>  1. Alter table to convert non-acid to acid.
>  - Do not allow this conversion on source of replication if strict.managed is 
> false.
> 2. Alter table or partition that changes the location.
>  - For managed tables at source, the table location shouldn't be changed for 
> the given non-partitioned table and partition location shouldn't be changed 
> for partitioned table as alter event doesn't capture the new files list. So, 
> it may cause data inconsistsency. So, if database is enabled for replication 
> at source, then alter location on managed tables should be blocked.
>  - For external partitioned tables, if location is changed at source, the the 
> location should be changed for the table and any partitions which reside 
> within the table location, but not for the partitions which are not within 
> the table location. (may be we just need the test).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)