[jira] [Commented] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796822#comment-16796822
 ] 

Hive QA commented on HIVE-21466:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
9s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 46s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16583/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16583/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.2.patch
>
>
> {code:java}
>  MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this 
> maximum is implemented, is that each block is added to the split, and if the 
> split grows to be larger than the maximum allowed, the split is submitted to 
> the cluster and a new split is opened.
> So, imagine the following scenario:
>  * HDFS block size of 16 bytes
>  * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block 

[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796802#comment-16796802
 ] 

Hive QA commented on HIVE-21422:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963006/HIVE-21422.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15832 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommands.testMergeOnTezEdges (batchId=327)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16582/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16582/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16582/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963006 - PreCommit-HIVE-Build

> Add metrics to LRFU cache policy
> 
>
> Key: HIVE-21422
> URL: https://issues.apache.org/jira/browse/HIVE-21422
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Affects Versions: 4.0.0
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
>  Labels: llap
> Fix For: 4.0.0
>
> Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch
>
>
> The LRFU cache policy for the LLAP data cache doesn't  provide enough insight 
> to figure out, what is cached and why something might get evicted. This 
> ticket is used to add Hadoop metrics 2 information (accessible via JMX) to 
> the LRFU policy, providing following information:
>  * How much memory is cached for data buffers
>  * How much memory is cached for meta data buffers
>  * How large is the min-heap of the cache policy
>  * How long is the eviction short list (linked list)
>  * How much memory is currently "locked" (buffers with positive reference 
> count) and therefore in use by a query
> These new counters are found in the MX bean, following this path:
> Hadoop/LlapDaemon/LowLevelLrfuCachePolicy-
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796780#comment-16796780
 ] 

Hive QA commented on HIVE-21422:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
46s{color} | {color:blue} llap-server in master has 79 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} llap-server: The patch generated 0 new + 0 unchanged 
- 24 fixed = 0 total (was 24) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 57s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16582/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: llap-server U: llap-server |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16582/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add metrics to LRFU cache policy
> 
>
> Key: HIVE-21422
> URL: https://issues.apache.org/jira/browse/HIVE-21422
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Affects Versions: 4.0.0
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
>  Labels: llap
> Fix For: 4.0.0
>
> Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch
>
>
> The LRFU cache policy for the LLAP data cache doesn't  provide enough insight 
> to figure out, what is cached and why something might get evicted. This 
> ticket is used to add Hadoop metrics 2 information (accessible via JMX) to 
> the LRFU policy, providing following information:
>  * How much memory is cached for data buffers
>  * How much memory is cached for meta data buffers
>  * How large is the min-heap of the cache policy
>  * How long is the eviction short list (linked list)
>  * How much memory is currently "locked" (buffers with positive reference 
> count) and therefore in use by a query
> These new counters are found in the MX bean, following this path:
> Hadoop/LlapDaemon/LowLevelLrfuCachePolicy-
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21409) Initial SessionState ClassLoader Reused For Subsequent Sessions

2019-03-19 Thread Shawn Weeks (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796783#comment-16796783
 ] 

Shawn Weeks commented on HIVE-21409:


I've spent a lot of time tracking this down and I've run into a dead end with a 
call to ReflectionUtils in HiveUtils.

getAuthorizerFactory. For some reason this call is changing the class loader 
for the current thread.

> Initial SessionState ClassLoader Reused For Subsequent Sessions
> ---
>
> Key: HIVE-21409
> URL: https://issues.apache.org/jira/browse/HIVE-21409
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Shawn Weeks
>Priority: Minor
> Attachments: create_class.sql, run.sql, setup.sql
>
>
> It appears that the first ClassLoader attached to a SessionState Static 
> Instance is being reused as the parent for all future sessions. This causes 
> any libraries added to the class path on the initial session to be added to 
> future sessions. It also appears that further sessions may be adding jars to 
> this initial ClassLoader as well leading to the class path getting more and 
> more polluted. This occurring on a build including HIVE-11878. I've included 
> some examples that greatly exaggerate the problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796771#comment-16796771
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963004/HIVE-21469.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 135 failed/errored test(s), 12413 tests 
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=267)
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out) 
(batchId=254)
TestBlobstoreCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=278)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed 
out) (batchId=254)
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=1)

[udf_upper.q,ctas_date.q,materialized_view_rewrite_part_2.q,groupby_grouping_sets3.q,vector_decimal_5.q,vector_case_when_conversion.q,bucket_map_join_spark4.q,timestamp_2.q,schema_evol_orc_acid_table_update_llap_io.q,date_join1.q,constprog_type.q,timestamp_ints_casts.q,udf_negative.q,orc_merge_diff_fs.q,udf_substring_index.q,results_cache_lifetime.q,cross_prod_3.q,masking_12.q,diff_part_input_formats.q,auto_join_without_localtask.q,join46.q,ctas_uses_table_location.q,tez_bmj_schema_evolution.q,bucketmapjoin4.q,udf_context_aware.q,authorization_non_id.q,mapjoin_test_outer.q,vectorization_9.q,input15.q,udf_PI.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=10)

[msck_repair_acid.q,parquet_ppd_decimal.q,authorization_load.q,udf_md5.q,alter_view_as_select.q,groupby_sort_6.q,limit0.q,select_as_omitted.q,decimal_udf.q,list_bucket_query_oneskew_3.q,vector_null_projection.q,smb_mapjoin_12.q,vector_cast_constant.q,encryption_insert_partition_static.q,semijoin3.q,reloadJar.q,orc_merge_incompat3.q,groupby2_limit.q,sort_merge_join_desc_1.q,vector_fullouter_mapjoin_1_optimized.q,ba_table3.q,mi.q,input0.q,parquet_map_of_arrays_of_ints.q,auto_join11.q,delete_whole_partition.q,authorization_view_disable_cbo_3.q,avro_date.q,udf_like.q,compute_stats_decimal.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=11)

[cbo_rp_lineage2.q,join_emit_interval.q,druidkafkamini_avro.q,mm_iow_temp.q,mapjoin2.q,schema_evol_text_vec_table_llap_io.q,vector_windowing_navfn.q,vectorization_12.q,vector_number_compare_projection.q,parquet_table_with_subschema.q,parquet_join2.q,authorization_8.q,exim_07_all_part_over_nonoverlap.q,parquet_ppd_char.q,udf_locate.q,nullgroup4_multi_distinct.q,join_rc.q,bucket6.q,schema_evol_text_vecrow_table.q,udf_likeany.q,orc_ppd_char.q,udf_boolean.q,udf_xpath_double.q,orc_ppd_same_table_multiple_aliases.q,materialized_view_drop.q,alter2.q,vector_full_outer_join.q,typechangetest.q,distinct_66.q,rcfile_merge4.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=12)

[skewjoinopt15.q,vector_coalesce.q,udf_elt.q,join44.q,udf5.q,udf_case_thrift.q,inputddl2.q,join_constraints_optimization.q,correlationoptimizer13.q,udf_testlength.q,parquet_vectorization_1.q,ppd_repeated_alias.q,correlationoptimizer4.q,ppd_windowing2.q,udf_union.q,masking_1_newdb.q,vector_leftsemi_mapjoin.q,udf_string.q,repl_1_drop.q,cbo_rp_insert.q,ppd_clusterby.q,udf_trunc_number.q,union10.q,vector_if_expr.q,load_dyn_part7.q,llap_acid2.q,groupby_constcolval.q,acid_mapjoin.q,explainanalyze_5.q,distinct_windowing.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=13)

[load_dyn_part3.q,autoColumnStats_4.q,drop_table.q,stat_estimate_drill.q,auto_join33.q,parquet_ppd_varchar.q,udf_sha2.q,groupby5_map_skew.q,merge4.q,storage_format_descriptor.q,mapjoin_hook.q,multi_column_in_single.q,schema_evol_orc_nonvec_table.q,cbo_rp_subq_in.q,authorization_view_disable_cbo_4.q,list_bucket_dml_2.q,cbo_rp_semijoin.q,char_2.q,non_ascii_literal2.q,load_part_authsuccess.q,auto_sortmerge_join_15.q,explain_rearrange.q,stats_nonpart.q,varchar_union1.q,vectorized_mapjoin3.q,input21.q,vector_udf2.q,groupby_cube_multi_gby.q,annotate_stats_limit.q,union34.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=14)

[auto_join18.q,partition_coltype_literals.q,input1_limit.q,udf_divide.q,input40.q,udf_round_2_auto_stats.q,correlationoptimizer8.q,auto_sortmerge_join_14.q,udf_array_contains.q,union22.q,non_ascii_literal1.q,bucket_map_join_tez2.q,sample_islocalmode_hook.q,literal_decimal.q,constprog2.q,parquet_external_time.q,infer_bucket_sort_reducers_power_two.q,input20.q,smb_join_partition_key.q,union_remove_14.q,vector_count.q,udf_if.q,input38.q,decimal_trailing.q,load_fs_overwrite.q,notable_alias2.q,join_reorder.q,constant_prop.q,sample1.q,bucketmapjoin8.q]
TestCliDriver - did not produce a TEST-*.xml 

[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications

2019-03-19 Thread mahesh kumar behera (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21446:
---
Status: Patch Available  (was: Open)

> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, 
> HIVE-21446.03.patch
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications

2019-03-19 Thread mahesh kumar behera (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21446:
---
Attachment: HIVE-21446.03.patch

> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, 
> HIVE-21446.03.patch
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications

2019-03-19 Thread mahesh kumar behera (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21446:
---
Status: Open  (was: Patch Available)

> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, 
> HIVE-21446.03.patch
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications

2019-03-19 Thread mahesh kumar behera (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-21446:
---
Attachment: (was: HIVE-21446.03.patch)

> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, 
> HIVE-21446.03.patch
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Work logged] (HIVE-21283) Create Synonym mid for substr, position for locate

2019-03-19 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21283?focusedWorklogId=215939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215939
 ]

ASF GitHub Bot logged work on HIVE-21283:
-

Author: ASF GitHub Bot
Created on: 20/Mar/19 01:38
Start Date: 20/Mar/19 01:38
Worklog Time Spent: 10m 
  Work Description: sankarh commented on issue #540: HIVE-21283 Synonyms 
for the existing functions
URL: https://github.com/apache/hive/pull/540#issuecomment-474649754
 
 
   @rmsmani 
   Yes, that is a problem with Hive ptest. Lot of flaky tests. 
   But, it was mandated to have a green build before commit. Please re-submit 
the same patch and try again.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 215939)
Time Spent: 2h 50m  (was: 2h 40m)

> Create Synonym mid for  substr, position for  locate
> 
>
> Key: HIVE-21283
> URL: https://issues.apache.org/jira/browse/HIVE-21283
> Project: Hive
>  Issue Type: New Feature
>Reporter: Mani M
>Assignee: Mani M
>Priority: Minor
>  Labels: UDF, pull-request-available, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, 
> HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, 
> HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.2.PATCH, 
> HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, 
> image-2019-03-16-21-33-18-898.png
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Create new synonym for the existing function
>  
> Mid for substr
> postiion for locate 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21456) Hive Metastore HTTP Thrift

2019-03-19 Thread Amit Khanna (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Khanna updated HIVE-21456:
---
Attachment: HIVE-21456.4.patch

> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Amit Khanna
>Priority: Major
> Attachments: HIVE-21456.2.patch, HIVE-21456.3.patch, 
> HIVE-21456.4.patch, HIVE-21456.patch
>
>
> Hive Metastore currently doesn't have support for HTTP transport because of 
> which it is not possible to access it via Knox. Adding support for Thrift 
> over HTTP transport will allow the clients to access via Knox



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21482) Partition discovery table property is added to non-partitioned external tables

2019-03-19 Thread Prasanth Jayachandran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-21482:
-
Attachment: HIVE-21482.1.patch

> Partition discovery table property is added to non-partitioned external tables
> --
>
> Key: HIVE-21482
> URL: https://issues.apache.org/jira/browse/HIVE-21482
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21482.1.patch
>
>
> Automatic partition discovery is added to external tables by default. But it 
> doesn't check if the external table is partitioned or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21482) Partition discovery table property is added to non-partitioned external tables

2019-03-19 Thread Prasanth Jayachandran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-21482:
-
Status: Patch Available  (was: Open)

> Partition discovery table property is added to non-partitioned external tables
> --
>
> Key: HIVE-21482
> URL: https://issues.apache.org/jira/browse/HIVE-21482
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21482.1.patch
>
>
> Automatic partition discovery is added to external tables by default. But it 
> doesn't check if the external table is partitioned or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21482) Partition discovery table property is added to non-partitioned external tables

2019-03-19 Thread Prasanth Jayachandran (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796663#comment-16796663
 ] 

Prasanth Jayachandran commented on HIVE-21482:
--

[~jdere] can you please review this patch? HIVE-20707 added partition discovery 
table property for all external tables without looking if the table is 
partitioned or not. This patch fixes it. 

> Partition discovery table property is added to non-partitioned external tables
> --
>
> Key: HIVE-21482
> URL: https://issues.apache.org/jira/browse/HIVE-21482
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21482.1.patch
>
>
> Automatic partition discovery is added to external tables by default. But it 
> doesn't check if the external table is partitioned or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796662#comment-16796662
 ] 

Hive QA commented on HIVE-21469:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
11s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} ql: The patch generated 0 new + 10 unchanged - 23 
fixed = 10 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16581/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16581/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
> 

[jira] [Commented] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler

2019-03-19 Thread Jesus Camacho Rodriguez (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796656#comment-16796656
 ] 

Jesus Camacho Rodriguez commented on HIVE-21468:


[~daijy], could you take a look? Thanks

> Case sensitivity in identifier names for JDBC storage handler
> -
>
> Key: HIVE-21468
> URL: https://issues.apache.org/jira/browse/HIVE-21468
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21468.01.patch, HIVE-21468.02.patch, 
> HIVE-21468.patch
>
>
> Currently, when Calcite generates the SQL query for the JDBC storage handler, 
> it will ignore capitalization for the identifiers names, which can lead to 
> errors at execution time (though the query is properly generated).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796652#comment-16796652
 ] 

Hive QA commented on HIVE-21468:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963002/HIVE-21468.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15833 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16580/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16580/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16580/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963002 - PreCommit-HIVE-Build

> Case sensitivity in identifier names for JDBC storage handler
> -
>
> Key: HIVE-21468
> URL: https://issues.apache.org/jira/browse/HIVE-21468
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21468.01.patch, HIVE-21468.02.patch, 
> HIVE-21468.patch
>
>
> Currently, when Calcite generates the SQL query for the JDBC storage handler, 
> it will ignore capitalization for the identifiers names, which can lead to 
> errors at execution time (though the query is properly generated).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Work logged] (HIVE-21283) Create Synonym mid for substr, position for locate

2019-03-19 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21283?focusedWorklogId=215908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215908
 ]

ASF GitHub Bot logged work on HIVE-21283:
-

Author: ASF GitHub Bot
Created on: 20/Mar/19 00:32
Start Date: 20/Mar/19 00:32
Worklog Time Spent: 10m 
  Work Description: rmsmani commented on issue #540: HIVE-21283 Synonyms 
for the existing functions
URL: https://github.com/apache/hive/pull/540#issuecomment-474636941
 
 
   Hi @sankarh
   Submitted patch 9 yday night, failed due to yet another Falky test
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 215908)
Time Spent: 2h 40m  (was: 2.5h)

> Create Synonym mid for  substr, position for  locate
> 
>
> Key: HIVE-21283
> URL: https://issues.apache.org/jira/browse/HIVE-21283
> Project: Hive
>  Issue Type: New Feature
>Reporter: Mani M
>Assignee: Mani M
>Priority: Minor
>  Labels: UDF, pull-request-available, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, 
> HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, 
> HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.2.PATCH, 
> HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, 
> image-2019-03-16-21-33-18-898.png
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Create new synonym for the existing function
>  
> Mid for substr
> postiion for locate 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796642#comment-16796642
 ] 

Hive QA commented on HIVE-21468:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
18s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
24s{color} | {color:blue} jdbc-handler in master has 12 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
42s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 0 fixed 
= 17 total (was 15) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
10s{color} | {color:red} jdbc-handler: The patch generated 3 new + 8 unchanged 
- 0 fixed = 11 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
27s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
33s{color} | {color:green} jdbc-handler generated 0 new + 11 unchanged - 1 
fixed = 11 total (was 12) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 26s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16580/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16580/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16580/yetus/diff-checkstyle-jdbc-handler.txt
 |
| modules | C: ql jdbc-handler itests U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16580/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Case sensitivity in identifier names for JDBC storage handler
> -
>
> Key: HIVE-21468
> URL: https://issues.apache.org/jira/browse/HIVE-21468
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21468.01.patch, HIVE-21468.02.patch, 
> HIVE-21468.patch
>
>
> Currently, when Calcite generates the SQL query for the 

[jira] [Assigned] (HIVE-21482) Partition discovery table property is added to non-partitioned external tables

2019-03-19 Thread Prasanth Jayachandran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-21482:



> Partition discovery table property is added to non-partitioned external tables
> --
>
> Key: HIVE-21482
> URL: https://issues.apache.org/jira/browse/HIVE-21482
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
>
> Automatic partition discovery is added to external tables by default. But it 
> doesn't check if the external table is partitioned or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796609#comment-16796609
 ] 

Hive QA commented on HIVE-21386:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962901/HIVE-21386.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 15832 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input13] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge_empty] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[null_column] (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nullformat] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge10] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge11] (batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge3] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge4] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge5] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge6] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge8] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge9] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat1] 
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat2] 
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat3] 
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat_schema]
 (batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat_writer_version]
 (batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[rename_external_partition_location]
 (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subq] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union] (batchId=4)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_bulk] 
(batchId=108)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge10] 
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge3] 
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge4] 
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[results_cache_diff_fs]
 (batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge11]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge5] 
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge6] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge7] 
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge8] 
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge9] 
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge_incompat1]
 (batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge_incompat2]
 (batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge_incompat3]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge_incompat_schema]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge_incompat_writer_version]
 (batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_orc_merge_incompat_schema]
 (batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge3]
 (batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge4]
 (batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge5]
 (batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge6]
 (batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge7]
 (batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge8]
 (batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge9]
 (batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat1]
 (batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat2]
 (batchId=193)

[jira] [Comment Edited] (HIVE-21480) Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup

2019-03-19 Thread Morio Ramdenbourg (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796607#comment-16796607
 ] 

Morio Ramdenbourg edited comment on HIVE-21480 at 3/19/19 11:15 PM:


Update: the current implementation of the test does nothing. The 
getJDOPersistenceManagerCacheSize method was always returning -1, since the 
fields in ObjectStore had been updated, and it could not grab the actual size 
of the PersistenceManagerCache.


was (Author: mramdenbourg):
Update: the test was testing nothing. The getJDOPersistenceManagerCacheSize 
method was always returning -1, since the fields in ObjectStore had been 
updated, and it could not grab the actual size of the PersistenceManagerCache.

> Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup
> ---
>
> Key: HIVE-21480
> URL: https://issues.apache.org/jira/browse/HIVE-21480
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
>
> [TestHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
>  tests whether the JDO persistence manager cache cleanup was performed 
> correctly when a HiveMetaStoreClient executes an API call, and closes. It 
> does this by ensuring that the cache object count before the API call, and 
> after closing, are the same. However, there are some assumptions that are not 
> always correct, and can cause flakiness.
> For example, lingering resources could be present from previous tests or from 
> setup depending on how PTest runs it, and can cause the object count to 
> sometimes be incorrect. We should rewrite this test to account for this 
> flakiness that can occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21480) Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup

2019-03-19 Thread Morio Ramdenbourg (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796607#comment-16796607
 ] 

Morio Ramdenbourg commented on HIVE-21480:
--

Update: the test was testing nothing. The getJDOPersistenceManagerCacheSize 
method was always returning -1, since the fields in ObjectStore had been 
updated, and it could not grab the actual size of the PersistenceManagerCache.

> Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup
> ---
>
> Key: HIVE-21480
> URL: https://issues.apache.org/jira/browse/HIVE-21480
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
>
> [TestHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
>  tests whether the JDO persistence manager cache cleanup was performed 
> correctly when a HiveMetaStoreClient executes an API call, and closes. It 
> does this by ensuring that the cache object count before the API call, and 
> after closing, are the same. However, there are some assumptions that are not 
> always correct, and can cause flakiness.
> For example, lingering resources could be present from previous tests or from 
> setup depending on how PTest runs it, and can cause the object count to 
> sometimes be incorrect. We should rewrite this test to account for this 
> flakiness that can occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

2019-03-19 Thread Deepak Jaiswal (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796587#comment-16796587
 ] 

Deepak Jaiswal commented on HIVE-21304:
---

[~vgarg] checkout OperatorFactory.java for details. The property propagates 
from the parent op. The code there is intentionally agnostic to operator type.

I believe there were other reasons too but I can't recall them.

> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796594#comment-16796594
 ] 

Hive QA commented on HIVE-21386:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
13s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
46s{color} | {color:red} ql: The patch generated 18 new + 615 unchanged - 0 
fixed = 633 total (was 615) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
34s{color} | {color:red} ql generated 1 new + 2254 unchanged - 1 fixed = 2255 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 57s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Dead store to fakePathExists in 
org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.setEntryValid(QueryResultsCache$CacheEntry,
 FetchWork)  At 
QueryResultsCache.java:org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.setEntryValid(QueryResultsCache$CacheEntry,
 FetchWork)  At QueryResultsCache.java:[line 551] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16579/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16579/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16579/yetus/new-findbugs-ql.html
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16579/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Extend the fetch task enhancement done in HIVE-21279 to make it work with 
> query result cache
> 
>
> Key: HIVE-21386
> URL: https://issues.apache.org/jira/browse/HIVE-21386
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21386.1.patch
>
>
> The improvement done in HIVE-21279 is disabled for query cache. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21480) Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup

2019-03-19 Thread Morio Ramdenbourg (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Morio Ramdenbourg updated HIVE-21480:
-
Description: 
[TestHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
 tests whether the JDO persistence manager cache cleanup was performed 
correctly when a HiveMetaStoreClient executes an API call, and closes. It does 
this by ensuring that the cache object count before the API call, and after 
closing, are the same. However, there are some assumptions that are not always 
correct, and can cause flakiness.

For example, lingering resources could be present from previous tests or from 
setup depending on how PTest runs it, and can cause the object count to 
sometimes be incorrect. We should rewrite this test to account for this 
flakiness that can occur.

  was:
[TestRemoteHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
 tests whether the JDO persistence manager cache cleanup was performed 
correctly when a HiveMetaStoreClient executes an API call, and closes. It does 
this by ensuring that the cache object count before the API call, and after 
closing, are the same. However, there are some assumptions that are not always 
correct, and can cause flakiness.

For example, lingering resources could be present from previous tests or from 
setup depending on how PTest runs it, and can cause the object count to 
sometimes be incorrect. We should rewrite this test to account for this 
flakiness that can occur.


> Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup
> ---
>
> Key: HIVE-21480
> URL: https://issues.apache.org/jira/browse/HIVE-21480
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
>
> [TestHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
>  tests whether the JDO persistence manager cache cleanup was performed 
> correctly when a HiveMetaStoreClient executes an API call, and closes. It 
> does this by ensuring that the cache object count before the API call, and 
> after closing, are the same. However, there are some assumptions that are not 
> always correct, and can cause flakiness.
> For example, lingering resources could be present from previous tests or from 
> setup depending on how PTest runs it, and can cause the object count to 
> sometimes be incorrect. We should rewrite this test to account for this 
> flakiness that can occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21480) Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup

2019-03-19 Thread Morio Ramdenbourg (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Morio Ramdenbourg updated HIVE-21480:
-
Summary: Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup  (was: 
Fix test TestRemoteHiveMetaStore.testJDOPersistanceManagerCleanup)

> Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup
> ---
>
> Key: HIVE-21480
> URL: https://issues.apache.org/jira/browse/HIVE-21480
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
>
> [TestRemoteHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
>  tests whether the JDO persistence manager cache cleanup was performed 
> correctly when a HiveMetaStoreClient executes an API call, and closes. It 
> does this by ensuring that the cache object count before the API call, and 
> after closing, are the same. However, there are some assumptions that are not 
> always correct, and can cause flakiness.
> For example, lingering resources could be present from previous tests or from 
> setup depending on how PTest runs it, and can cause the object count to 
> sometimes be incorrect. We should rewrite this test to account for this 
> flakiness that can occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Work started] (HIVE-21480) Fix test TestRemoteHiveMetaStore.testJDOPersistanceManagerCleanup

2019-03-19 Thread Morio Ramdenbourg (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-21480 started by Morio Ramdenbourg.

> Fix test TestRemoteHiveMetaStore.testJDOPersistanceManagerCleanup
> -
>
> Key: HIVE-21480
> URL: https://issues.apache.org/jira/browse/HIVE-21480
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
>
> [TestRemoteHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
>  tests whether the JDO persistence manager cache cleanup was performed 
> correctly when a HiveMetaStoreClient executes an API call, and closes. It 
> does this by ensuring that the cache object count before the API call, and 
> after closing, are the same. However, there are some assumptions that are not 
> always correct, and can cause flakiness.
> For example, lingering resources could be present from previous tests or from 
> setup depending on how PTest runs it, and can cause the object count to 
> sometimes be incorrect. We should rewrite this test to account for this 
> flakiness that can occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-21480) Fix test TestRemoteHiveMetaStore.testJDOPersistanceManagerCleanup

2019-03-19 Thread Morio Ramdenbourg (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Morio Ramdenbourg reassigned HIVE-21480:



> Fix test TestRemoteHiveMetaStore.testJDOPersistanceManagerCleanup
> -
>
> Key: HIVE-21480
> URL: https://issues.apache.org/jira/browse/HIVE-21480
> Project: Hive
>  Issue Type: Improvement
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
>
> [TestRemoteHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
>  tests whether the JDO persistence manager cache cleanup was performed 
> correctly when a HiveMetaStoreClient executes an API call, and closes. It 
> does this by ensuring that the cache object count before the API call, and 
> after closing, are the same. However, there are some assumptions that are not 
> always correct, and can cause flakiness.
> For example, lingering resources could be present from previous tests or from 
> setup depending on how PTest runs it, and can cause the object count to 
> sometimes be incorrect. We should rewrite this test to account for this 
> flakiness that can occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21034) Add option to schematool to drop Hive databases

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796568#comment-16796568
 ] 

Hive QA commented on HIVE-21034:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962986/HIVE-21034.5.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15836 tests 
executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testCancelRenewTokenFlow 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testConnection 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testIsValid (batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testIsValidNeg 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNegativeProxyAuth 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNegativeTokenAuth 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNoKrbSASLTokenAuthNeg 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNonKrbSASLAuth 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNonKrbSASLFullNameAuth
 (batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testProxyAuth 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testRenewDelegationToken 
(batchId=276)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testTokenAuth 
(batchId=276)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16578/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16578/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16578/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962986 - PreCommit-HIVE-Build

> Add option to schematool to drop Hive databases
> ---
>
> Key: HIVE-21034
> URL: https://issues.apache.org/jira/browse/HIVE-21034
> Project: Hive
>  Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21034.1.patch, HIVE-21034.2.patch, 
> HIVE-21034.2.patch, HIVE-21034.3.patch, HIVE-21034.4.patch, HIVE-21034.5.patch
>
>
> An option to remove all Hive managed data could be a useful addition to 
> {{schematool}}.
> I propose to introduce a new flag {{-dropAllDatabases}} that would *drop all 
> databases with CASCADE* to remove all data of managed tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21034) Add option to schematool to drop Hive databases

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796531#comment-16796531
 ] 

Hive QA commented on HIVE-21034:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m 
16s{color} | {color:blue} standalone-metastore/metastore-server in master has 
179 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 48s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16578/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U: 
standalone-metastore/metastore-server |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16578/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add option to schematool to drop Hive databases
> ---
>
> Key: HIVE-21034
> URL: https://issues.apache.org/jira/browse/HIVE-21034
> Project: Hive
>  Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21034.1.patch, HIVE-21034.2.patch, 
> HIVE-21034.2.patch, HIVE-21034.3.patch, HIVE-21034.4.patch, HIVE-21034.5.patch
>
>
> An option to remove all Hive managed data could be a useful addition to 
> {{schematool}}.
> I propose to introduce a new flag {{-dropAllDatabases}} that would *drop all 
> databases with CASCADE* to remove all data of managed tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21474) Bumping guava version

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796510#comment-16796510
 ] 

Hive QA commented on HIVE-21474:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962983/HIVE-21474.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 15826 tests 
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=267)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=267)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_with_join2] 
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=54)
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles (batchId=331)
org.apache.hive.spark.client.TestSparkClient.testCounters (batchId=331)
org.apache.hive.spark.client.TestSparkClient.testErrorJob (batchId=331)
org.apache.hive.spark.client.TestSparkClient.testErrorJobNotSerializable 
(batchId=331)
org.apache.hive.spark.client.TestSparkClient.testJobSubmission (batchId=331)
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection (batchId=331)
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob (batchId=331)
org.apache.hive.spark.client.TestSparkClient.testSyncRpc (batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testAutoRegistration 
(batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testDecryptionOnly 
(batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testEmbeddedChannel 
(batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testEncryptDecrypt 
(batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testEncryptionOnly 
(batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testFragmentation 
(batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testKryoCodec 
(batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testMaxMessageSize 
(batchId=331)
org.apache.hive.spark.client.rpc.TestKryoMessageCodec.testNegativeMessageSize 
(batchId=331)
org.apache.hive.spark.client.rpc.TestRpc.testBadHello (batchId=331)
org.apache.hive.spark.client.rpc.TestRpc.testClientServer (batchId=331)
org.apache.hive.spark.client.rpc.TestRpc.testCloseListener (batchId=331)
org.apache.hive.spark.client.rpc.TestRpc.testEncryption (batchId=331)
org.apache.hive.spark.client.rpc.TestRpc.testNotDeserializableRpc (batchId=331)
org.apache.hive.spark.client.rpc.TestRpc.testRpcDispatcher (batchId=331)
org.apache.hive.spark.client.rpc.TestRpc.testRpcServerMultiThread (batchId=331)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16577/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16577/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16577/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962983 - PreCommit-HIVE-Build

> Bumping guava version
> -
>
> Key: HIVE-21474
> URL: https://issues.apache.org/jira/browse/HIVE-21474
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21474.patch
>
>
> Bump guava to 24.1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21474) Bumping guava version

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796506#comment-16796506
 ] 

Hive QA commented on HIVE-21474:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
14s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
32s{color} | {color:blue} druid-handler in master has 3 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
31s{color} | {color:red} ql generated 3 new + 2244 unchanged - 11 fixed = 2247 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Null passed for non-null parameter of addJars(String) in 
org.apache.hadoop.hive.ql.exec.spark.LocalHiveSparkClient.refreshLocalResources(SparkWork,
 HiveConf)  Method invoked at LocalHiveSparkClient.java:of addJars(String) in 
org.apache.hadoop.hive.ql.exec.spark.LocalHiveSparkClient.refreshLocalResources(SparkWork,
 HiveConf)  Method invoked at LocalHiveSparkClient.java:[line 195] |
|  |  Null passed for non-null parameter of addJars(String) in 
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.refreshLocalResources(SparkWork,
 HiveConf)  Method invoked at RemoteHiveSparkClient.java:of addJars(String) in 
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.refreshLocalResources(SparkWork,
 HiveConf)  Method invoked at RemoteHiveSparkClient.java:[line 238] |
|  |  Null passed for non-null parameter of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.processCurrentEvents(WorkloadManager$EventState,
 WorkloadManager$WmThreadSyncWork)  At WorkloadManager.java:of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.processCurrentEvents(WorkloadManager$EventState,
 WorkloadManager$WmThreadSyncWork)  At WorkloadManager.java:[line 733] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16577/dev-support/hive-personality.sh
 |
| git 

[jira] [Commented] (HIVE-21473) Bumping jackson version

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796451#comment-16796451
 ] 

Hive QA commented on HIVE-21473:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962977/HIVE-21473.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15832 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16576/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16576/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16576/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962977 - PreCommit-HIVE-Build

> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21473.patch
>
>
> Bump jackson version to 2.9.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21473) Bumping jackson version

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796430#comment-16796430
 ] 

Hive QA commented on HIVE-21473:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 43s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16576/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| modules | C: . U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16576/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21473.patch
>
>
> Bump jackson version to 2.9.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21283) Create Synonym mid for substr, position for locate

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796371#comment-16796371
 ] 

Hive QA commented on HIVE-21283:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962958/HIVE.21283.09.PATCH

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15834 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce] 
(batchId=61)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16575/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16575/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16575/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962958 - PreCommit-HIVE-Build

> Create Synonym mid for  substr, position for  locate
> 
>
> Key: HIVE-21283
> URL: https://issues.apache.org/jira/browse/HIVE-21283
> Project: Hive
>  Issue Type: New Feature
>Reporter: Mani M
>Assignee: Mani M
>Priority: Minor
>  Labels: UDF, pull-request-available, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, 
> HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, 
> HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.2.PATCH, 
> HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, 
> image-2019-03-16-21-33-18-898.png
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Create new synonym for the existing function
>  
> Mid for substr
> postiion for locate 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-17404) Orc split generation cache does not handle files without file tail

2019-03-19 Thread Prasanth Jayachandran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17404:
-
Attachment: HIVE-17404.2.patch

> Orc split generation cache does not handle files without file tail
> --
>
> Key: HIVE-17404
> URL: https://issues.apache.org/jira/browse/HIVE-17404
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Prasanth Jayachandran
>Assignee: Aditya Shah
>Priority: Critical
> Attachments: HIVE-17404.2.patch, HIVE-17404.patch
>
>
> Some old files do not have Orc FileTail. If file tail does not exist, split 
> generation should fallback to old way of storing footers. 
> This can result in exceptions like below
> {code}
> ORC split generation failed with exception: Malformed ORC file. Invalid 
> postscript length 9
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1735)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1822)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:450)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:569)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
>   at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
>   at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
>   at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
>   at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.orc.FileFormatException: Malformed ORC file. Invalid 
> postscript length 9
>   at org.apache.orc.impl.ReaderImpl.ensureOrcFooter(ReaderImpl.java:297)
>   at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:470)
>   at 
> org.apache.hadoop.hive.ql.io.orc.LocalCache.getAndValidate(LocalCache.java:103)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.getSplits(OrcInputFormat.java:804)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.runGetSplitsSync(OrcInputFormat.java:922)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.generateSplitWork(OrcInputFormat.java:891)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.scheduleSplits(OrcInputFormat.java:1763)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1707)
>   ... 15 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-17404) Orc split generation cache does not handle files without file tail

2019-03-19 Thread Prasanth Jayachandran (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796372#comment-16796372
 ] 

Prasanth Jayachandran commented on HIVE-17404:
--

+1

> Orc split generation cache does not handle files without file tail
> --
>
> Key: HIVE-17404
> URL: https://issues.apache.org/jira/browse/HIVE-17404
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Prasanth Jayachandran
>Assignee: Aditya Shah
>Priority: Critical
> Attachments: HIVE-17404.2.patch, HIVE-17404.patch
>
>
> Some old files do not have Orc FileTail. If file tail does not exist, split 
> generation should fallback to old way of storing footers. 
> This can result in exceptions like below
> {code}
> ORC split generation failed with exception: Malformed ORC file. Invalid 
> postscript length 9
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1735)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1822)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:450)
>   at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:569)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
>   at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
>   at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
>   at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
>   at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.orc.FileFormatException: Malformed ORC file. Invalid 
> postscript length 9
>   at org.apache.orc.impl.ReaderImpl.ensureOrcFooter(ReaderImpl.java:297)
>   at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:470)
>   at 
> org.apache.hadoop.hive.ql.io.orc.LocalCache.getAndValidate(LocalCache.java:103)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.getSplits(OrcInputFormat.java:804)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.runGetSplitsSync(OrcInputFormat.java:922)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.generateSplitWork(OrcInputFormat.java:891)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.scheduleSplits(OrcInputFormat.java:1763)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1707)
>   ... 15 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21460) ACID: Load data followed by a select * query results in incorrect results

2019-03-19 Thread Gopal V (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796350#comment-16796350
 ] 

Gopal V commented on HIVE-21460:


LGTM - +1 

(backport to 3.x branch too)

> ACID: Load data followed by a select * query results in incorrect results
> -
>
> Key: HIVE-21460
> URL: https://issues.apache.org/jira/browse/HIVE-21460
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Blocker
> Attachments: HIVE-21460.1.patch
>
>
> This affects current master as well. Created an orc file such that it spans 
> multiple stripes and ran a simple select *, and got incorrect row counts 
> (when comparing with select count(*). The problem seems to be that after 
> split generation and creating min/max rowId for each row (note that since the 
> loaded file is not written by Hive ACID, it does not have ROW__ID in the 
> file; but the ROW__ID is applied on read by discovering min/max bounds which 
> are used for calculating ROW__ID.rowId for each row of a split), Hive is only 
> reading the last split.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21283) Create Synonym mid for substr, position for locate

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796313#comment-16796313
 ] 

Hive QA commented on HIVE-21283:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
 1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
15s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 15s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16575/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16575/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Create Synonym mid for  substr, position for  locate
> 
>
> Key: HIVE-21283
> URL: https://issues.apache.org/jira/browse/HIVE-21283
> Project: Hive
>  Issue Type: New Feature
>Reporter: Mani M
>Assignee: Mani M
>Priority: Minor
>  Labels: UDF, pull-request-available, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, 
> HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, 
> HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.2.PATCH, 
> HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, 
> image-2019-03-16-21-33-18-898.png
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Create new synonym for the existing function
>  
> Mid for substr
> postiion for locate 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21479) NPE during metastore cache update

2019-03-19 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21479:
--
Attachment: HIVE-21479.1.patch

> NPE during metastore cache update
> -
>
> Key: HIVE-21479
> URL: https://issues.apache.org/jira/browse/HIVE-21479
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21479.1.patch
>
>
> Saw the following stack during a long periodical update:
> {code}
> 2019-03-12T10:01:43,015 ERROR [CachedStore-CacheUpdateService: Thread-36] 
> cache.CachedStore: Update failure:java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.updateTableColStats(CachedStore.java:508)
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.update(CachedStore.java:461)
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:396)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> The reason is we get the table list at very early stage and then refresh 
> table one by one. It is likely table is removed during the interim. We need 
> to deal with this case during cache update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21479) NPE during metastore cache update

2019-03-19 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21479:
--
Status: Patch Available  (was: Open)

> NPE during metastore cache update
> -
>
> Key: HIVE-21479
> URL: https://issues.apache.org/jira/browse/HIVE-21479
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21479.1.patch
>
>
> Saw the following stack during a long periodical update:
> {code}
> 2019-03-12T10:01:43,015 ERROR [CachedStore-CacheUpdateService: Thread-36] 
> cache.CachedStore: Update failure:java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.updateTableColStats(CachedStore.java:508)
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.update(CachedStore.java:461)
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:396)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> The reason is we get the table list at very early stage and then refresh 
> table one by one. It is likely table is removed during the interim. We need 
> to deal with this case during cache update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-21479) NPE during metastore cache update

2019-03-19 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-21479:
-


> NPE during metastore cache update
> -
>
> Key: HIVE-21479
> URL: https://issues.apache.org/jira/browse/HIVE-21479
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
>
> Saw the following stack during a long periodical update:
> {code}
> 2019-03-12T10:01:43,015 ERROR [CachedStore-CacheUpdateService: Thread-36] 
> cache.CachedStore: Update failure:java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.updateTableColStats(CachedStore.java:508)
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.update(CachedStore.java:461)
>   at 
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:396)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> The reason is we get the table list at very early stage and then refresh 
> table one by one. It is likely table is removed during the interim. We need 
> to deal with this case during cache update.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-21478) Metastore cache update shall capture exception

2019-03-19 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-21478:
-


> Metastore cache update shall capture exception
> --
>
> Key: HIVE-21478
> URL: https://issues.apache.org/jira/browse/HIVE-21478
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21478.1.patch
>
>
> We definitely need to capture any exception during 
> CacheUpdateMasterWork.update(), otherwise, Java would refuse to schedule 
> future update().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21478) Metastore cache update shall capture exception

2019-03-19 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21478:
--
Attachment: HIVE-21478.1.patch

> Metastore cache update shall capture exception
> --
>
> Key: HIVE-21478
> URL: https://issues.apache.org/jira/browse/HIVE-21478
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21478.1.patch
>
>
> We definitely need to capture any exception during 
> CacheUpdateMasterWork.update(), otherwise, Java would refuse to schedule 
> future update().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21478) Metastore cache update shall capture exception

2019-03-19 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-21478:
--
Status: Patch Available  (was: Open)

> Metastore cache update shall capture exception
> --
>
> Key: HIVE-21478
> URL: https://issues.apache.org/jira/browse/HIVE-21478
> Project: Hive
>  Issue Type: Bug
>  Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21478.1.patch
>
>
> We definitely need to capture any exception during 
> CacheUpdateMasterWork.update(), otherwise, Java would refuse to schedule 
> future update().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21455:
--
Status: Patch Available  (was: Open)

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 3.1.0, 3.0.0, 2.1.0, 2.0.0, 1.1.0, 1.2.0
>Reporter: Miklos Szurap
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21455:
--
Attachment: HIVE-21455.2.patch

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor reassigned HIVE-21455:
-

Assignee: Miklos Szurap  (was: David Mollitor)

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21455:
--
Status: Open  (was: Patch Available)

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 3.1.0, 3.0.0, 2.1.0, 2.0.0, 1.1.0, 1.2.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor reassigned HIVE-21455:
-

Assignee: David Mollitor  (was: Miklos Szurap)

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO  
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader: 
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job: 
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema 
> and thousands of datafiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy

2019-03-19 Thread Oliver Draese (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796256#comment-16796256
 ] 

Oliver Draese commented on HIVE-21422:
--

Added a patch version (2) which removes all checkstyle warnings.

> Add metrics to LRFU cache policy
> 
>
> Key: HIVE-21422
> URL: https://issues.apache.org/jira/browse/HIVE-21422
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Affects Versions: 4.0.0
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
>  Labels: llap
> Fix For: 4.0.0
>
> Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch
>
>
> The LRFU cache policy for the LLAP data cache doesn't  provide enough insight 
> to figure out, what is cached and why something might get evicted. This 
> ticket is used to add Hadoop metrics 2 information (accessible via JMX) to 
> the LRFU policy, providing following information:
>  * How much memory is cached for data buffers
>  * How much memory is cached for meta data buffers
>  * How large is the min-heap of the cache policy
>  * How long is the eviction short list (linked list)
>  * How much memory is currently "locked" (buffers with positive reference 
> count) and therefore in use by a query
> These new counters are found in the MX bean, following this path:
> Hadoop/LlapDaemon/LowLevelLrfuCachePolicy-
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796269#comment-16796269
 ] 

Hive QA commented on HIVE-21304:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962955/HIVE-21304.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 241 failed/errored test(s), 15833 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11] 
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12] 
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13] 
(batchId=275)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
 (batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
 (batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
 (batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
 (batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
 (batchId=278)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5a] 
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8] 
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_reordering_values]
 (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[binary_output_format] 
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket1] (batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket2] (batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket3] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_1] 
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_2] 
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1] 
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_1] 
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_2] 
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_3] 
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_4] 
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_5] 
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_6] 
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_7] 
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_8] 
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin10] 
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin11] 
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12] 
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin13] 
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5] 
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin8] 
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9] 
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative2] 
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative] 
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_outer_join_ppr] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl] 
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_tbllvl] 
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantPropagateForSubQuery]
 (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cp_sel] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[display_colstats_tbllvl] 
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_aggr] (batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_join_breaktask] 
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_union] 

[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21466:
--
Status: Patch Available  (was: Open)

> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.2.patch
>
>
> {code:java}
>  MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this 
> maximum is implemented, is that each block is added to the split, and if the 
> split grows to be larger than the maximum allowed, the split is submitted to 
> the cluster and a new split is opened.
> So, imagine the following scenario:
>  * HDFS block size of 16 bytes
>  * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be 
> inserted, (3x16) = 48 bytes in the split. So, while many operators would 
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split 
> size to a multiple of the HDFS block size will make this behavior less 
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS 
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes 
> are 128MB by default, so I propose setting this to 4x128MB.  The larger 
> splits (fewer tasks) should give a nice performance boost for modern hardware.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21466:
--
Status: Open  (was: Patch Available)

> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.2.patch
>
>
> {code:java}
>  MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this 
> maximum is implemented, is that each block is added to the split, and if the 
> split grows to be larger than the maximum allowed, the split is submitted to 
> the cluster and a new split is opened.
> So, imagine the following scenario:
>  * HDFS block size of 16 bytes
>  * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be 
> inserted, (3x16) = 48 bytes in the split. So, while many operators would 
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split 
> size to a multiple of the HDFS block size will make this behavior less 
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS 
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes 
> are 128MB by default, so I propose setting this to 4x128MB.  The larger 
> splits (fewer tasks) should give a nice performance boost for modern hardware.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21466:
--
Attachment: (was: HIVE-21466.1.patch)

> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.2.patch
>
>
> {code:java}
>  MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this 
> maximum is implemented, is that each block is added to the split, and if the 
> split grows to be larger than the maximum allowed, the split is submitted to 
> the cluster and a new split is opened.
> So, imagine the following scenario:
>  * HDFS block size of 16 bytes
>  * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be 
> inserted, (3x16) = 48 bytes in the split. So, while many operators would 
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split 
> size to a multiple of the HDFS block size will make this behavior less 
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS 
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes 
> are 128MB by default, so I propose setting this to 4x128MB.  The larger 
> splits (fewer tasks) should give a nice performance boost for modern hardware.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21466:
--
Attachment: HIVE-21466.2.patch

> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch, HIVE-21466.2.patch
>
>
> {code:java}
>  MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this 
> maximum is implemented, is that each block is added to the split, and if the 
> split grows to be larger than the maximum allowed, the split is submitted to 
> the cluster and a new split is opened.
> So, imagine the following scenario:
>  * HDFS block size of 16 bytes
>  * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be 
> inserted, (3x16) = 48 bytes in the split. So, while many operators would 
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split 
> size to a multiple of the HDFS block size will make this behavior less 
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS 
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes 
> are 128MB by default, so I propose setting this to 4x128MB.  The larger 
> splits (fewer tasks) should give a nice performance boost for modern hardware.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy

2019-03-19 Thread Oliver Draese (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oliver Draese updated HIVE-21422:
-
Attachment: HIVE-21422.2.patch

> Add metrics to LRFU cache policy
> 
>
> Key: HIVE-21422
> URL: https://issues.apache.org/jira/browse/HIVE-21422
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Affects Versions: 4.0.0
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
>  Labels: llap
> Fix For: 4.0.0
>
> Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch
>
>
> The LRFU cache policy for the LLAP data cache doesn't  provide enough insight 
> to figure out, what is cached and why something might get evicted. This 
> ticket is used to add Hadoop metrics 2 information (accessible via JMX) to 
> the LRFU policy, providing following information:
>  * How much memory is cached for data buffers
>  * How much memory is cached for meta data buffers
>  * How large is the min-heap of the cache policy
>  * How long is the eviction short list (linked list)
>  * How much memory is currently "locked" (buffers with positive reference 
> count) and therefore in use by a query
> These new counters are found in the MX bean, following this path:
> Hadoop/LlapDaemon/LowLevelLrfuCachePolicy-
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21469:
--
Status: Patch Available  (was: Open)

> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
>   }
> {code}
> There has historically been a quite a few bugs regarding leaked locks.  The 
> Driver will signal the session {{Thread}} by performing an interrupt.  That 
> interrupt can happen any time and it can kill a create/delete action within 
> the ZK framework.  We can see one example of workaround for this.  If the ZK 
> action is interrupted, simply do it again.  Well, what if it's interrupted 
> yet again?  The lock will be leaked.  Also, when the {{InterruptedException}} 
> is caught in the try block, the thread's interrupted flag is cleared.  The 
> flag is not reset in this code and therefore we lose the fact that this 
> thread has been interrupted.  This flag should be preserved so that other 
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
>   Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
>   } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
>   String name = ((ZooKeeperHiveLock)hiveLock).getPath();
>   throw new LockException("Node " + name + " can not be deleted after 
> " + numRetriesForUnLock + " attempts.",
>   e);
> }
>   }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete 
> the lock (again, for fear of leaking it).  This sleep should be 
> uninterruptible.  If we need to get the lock deleted, and there's a problem, 
> interrupting the sleep will cause the code to eventually exit and locks will 
> be leaked.
> It also requires a bunch more TLC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21469:
--
Attachment: HIVE-21469.2.patch

> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
>   }
> {code}
> There has historically been a quite a few bugs regarding leaked locks.  The 
> Driver will signal the session {{Thread}} by performing an interrupt.  That 
> interrupt can happen any time and it can kill a create/delete action within 
> the ZK framework.  We can see one example of workaround for this.  If the ZK 
> action is interrupted, simply do it again.  Well, what if it's interrupted 
> yet again?  The lock will be leaked.  Also, when the {{InterruptedException}} 
> is caught in the try block, the thread's interrupted flag is cleared.  The 
> flag is not reset in this code and therefore we lose the fact that this 
> thread has been interrupted.  This flag should be preserved so that other 
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
>   Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
>   } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
>   String name = ((ZooKeeperHiveLock)hiveLock).getPath();
>   throw new LockException("Node " + name + " can not be deleted after 
> " + numRetriesForUnLock + " attempts.",
>   e);
> }
>   }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete 
> the lock (again, for fear of leaking it).  This sleep should be 
> uninterruptible.  If we need to get the lock deleted, and there's a problem, 
> interrupting the sleep will cause the code to eventually exit and locks will 
> be leaked.
> It also requires a bunch more TLC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler

2019-03-19 Thread Jesus Camacho Rodriguez (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-21468:
---
Attachment: HIVE-21468.02.patch

> Case sensitivity in identifier names for JDBC storage handler
> -
>
> Key: HIVE-21468
> URL: https://issues.apache.org/jira/browse/HIVE-21468
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21468.01.patch, HIVE-21468.02.patch, 
> HIVE-21468.patch
>
>
> Currently, when Calcite generates the SQL query for the JDBC storage handler, 
> it will ignore capitalization for the identifiers names, which can lead to 
> errors at execution time (though the query is properly generated).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796230#comment-16796230
 ] 

Hive QA commented on HIVE-21304:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
29s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
49s{color} | {color:red} ql: The patch generated 2 new + 989 unchanged - 3 
fixed = 991 total (was 992) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 27s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16574/dev-support/hive-personality.sh
 |
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16574/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16574/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By

2019-03-19 Thread Ashutosh Chauhan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796214#comment-16796214
 ] 

Ashutosh Chauhan commented on HIVE-16924:
-

please remove ql/src/test/queries/clientnegative/udaf_invalid_place.q too, its 
not needed anymore.

> Support distinct in presence of Group By 
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch, 
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch, 
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch, 
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch, 
> HIVE-16924.12.patch, HIVE-16924.13.patch, HIVE-16924.14.patch, 
> HIVE-16924.15.patch, HIVE-16924.16.patch, HIVE-16924.17.patch, 
> HIVE-16924.18.patch, HIVE-16924.19.patch, HIVE-16924.20.patch, 
> HIVE-16924.21.patch, HIVE-16924.22.patch, HIVE-16924.23.patch, 
> HIVE-16924.24.patch, HIVE-16924.25.patch, HIVE-16924.26.patch, 
> HIVE-16924.27.patch
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get : 
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the 
> same query. Error encountered near token 'c1'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache

2019-03-19 Thread Vineet Garg (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-21386:
---
Status: Patch Available  (was: Open)

> Extend the fetch task enhancement done in HIVE-21279 to make it work with 
> query result cache
> 
>
> Key: HIVE-21386
> URL: https://issues.apache.org/jira/browse/HIVE-21386
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21386.1.patch
>
>
> The improvement done in HIVE-21279 is disabled for query cache. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

2019-03-19 Thread Vineet Garg (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796213#comment-16796213
 ] 

Vineet Garg commented on HIVE-21304:


Curious on why is {{bucketingVersion}} part of {{Operator/OperatorDesc}}. Since 
it is RS specific shouldn't it be part of only {{ReduceSinkDesc}}?

> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21455) Too verbose logging in AvroGenericRecordReader

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796200#comment-16796200
 ] 

Hive QA commented on HIVE-21455:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962836/HIVE-21455.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16573/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16573/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16573/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-19 15:28:47.035
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16573/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-19 15:28:47.038
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 36bd89d HIVE-16924 : Support distinct in presence of Group By 
(Miklos Gergely via Zoltan Haindrich)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 36bd89d HIVE-16924 : Support distinct in presence of Group By 
(Miklos Gergely via Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-19 15:28:47.700
+ rm -rf ../yetus_PreCommit-HIVE-Build-16573
+ mkdir ../yetus_PreCommit-HIVE-Build-16573
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16573
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16573/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java: 
does not exist in index
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java:134
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java' 
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java:134
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java' 
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16573
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962836 - PreCommit-HIVE-Build

> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too 
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the 
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO  org.apache.hadoop.mapred.FileInputFormat: 
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 

[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By

2019-03-19 Thread Jesus Camacho Rodriguez (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796194#comment-16796194
 ] 

Jesus Camacho Rodriguez commented on HIVE-16924:


[~ashutoshc], [~mgergely], this patch deleted 
{{ql/src/test/results/clientnegative/udaf_invalid_place.q.out}}, which causes 
test to fail. If this was intentional, we need to delete 
{{ql/src/test/queries/clientnegative/udaf_invalid_place.q}} too. If it was not, 
we need to regenerate the {{q.out}} file. Could you push an addendum or let me 
know? Thanks

> Support distinct in presence of Group By 
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch, 
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch, 
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch, 
> HIVE-16924.09.patch, HIVE-16924.10.patch, HIVE-16924.11.patch, 
> HIVE-16924.12.patch, HIVE-16924.13.patch, HIVE-16924.14.patch, 
> HIVE-16924.15.patch, HIVE-16924.16.patch, HIVE-16924.17.patch, 
> HIVE-16924.18.patch, HIVE-16924.19.patch, HIVE-16924.20.patch, 
> HIVE-16924.21.patch, HIVE-16924.22.patch, HIVE-16924.23.patch, 
> HIVE-16924.24.patch, HIVE-16924.25.patch, HIVE-16924.26.patch, 
> HIVE-16924.27.patch
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get : 
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the 
> same query. Error encountered near token 'c1'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796189#comment-16796189
 ] 

Hive QA commented on HIVE-21446:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962948/HIVE-21446.03.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15814 tests 
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=275)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=275)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=275)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
 (batchId=99)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics
 (batchId=234)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16572/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16572/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16572/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962948 - PreCommit-HIVE-Build

> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, 
> HIVE-21446.03.patch
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

2019-03-19 Thread Deepak Jaiswal (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796181#comment-16796181
 ] 

Deepak Jaiswal commented on HIVE-21304:
---

[~kgyrtkirk] thanks for taking this up. I don't mind.

I looked at the patch. Hope it runs clean. For some reason, I don't remember 
why I put this in the Operator instead of Conf. I do remember putting it in 
conf and facing problems due to it.

Please let me know if you get clean run for a code review.

> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (HIVE-21475) SparkClientUtilities::urlFromPathString should handle viewfs to avoid UDF ClassNotFoundExcpetion

2019-03-19 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan resolved HIVE-21475.
-
Resolution: Duplicate

Already fixed in master. Closing this as duplicate of HIVE-16292.

> SparkClientUtilities::urlFromPathString should handle viewfs to avoid UDF 
> ClassNotFoundExcpetion
> 
>
> Key: HIVE-21475
> URL: https://issues.apache.org/jira/browse/HIVE-21475
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Trivial
>
> e.g error
> {noformat}
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> invertedWorkGraph (org.apache.hadoop.hive.ql.plan.SparkWork)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:181)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
>   at 
> 

[jira] [Updated] (HIVE-21475) SparkClientUtilities::urlFromPathString should handle viewfs to avoid UDF ClassNotFoundExcpetion

2019-03-19 Thread Rajesh Balamohan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-21475:

Description: 
e.g error

{noformat}

childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
invertedWorkGraph (org.apache.hadoop.hive.ql.plan.SparkWork)
at 
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
at 
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:181)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
at 
org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:206)
at 
org.apache.hadoop.hive.ql.exec.spark.KryoSerializer.deserialize(KryoSerializer.java:60)
{noformat}

> SparkClientUtilities::urlFromPathString should handle viewfs to avoid UDF 
> 

[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796145#comment-16796145
 ] 

Hive QA commented on HIVE-21446:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
18s{color} | {color:blue} shims/common in master has 6 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
25s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
10s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} shims/common: The patch generated 0 new + 94 
unchanged - 1 fixed = 94 total (was 95) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} The patch 0.23 passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 2 new + 19 unchanged - 2 fixed 
= 21 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16572/dev-support/hive-personality.sh
 |
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16572/yetus/diff-checkstyle-ql.txt
 |
| modules | C: shims/common shims/0.23 common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16572/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
>

[jira] [Commented] (HIVE-21474) Bumping guava version

2019-03-19 Thread Zoltan Haindrich (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796128#comment-16796128
 ] 

Zoltan Haindrich commented on HIVE-21474:
-

+1 pending tests

> Bumping guava version
> -
>
> Key: HIVE-21474
> URL: https://issues.apache.org/jira/browse/HIVE-21474
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21474.patch
>
>
> Bump guava to 24.1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager

2019-03-19 Thread David Mollitor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Mollitor updated HIVE-21469:
--
Status: Open  (was: Patch Available)

> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
>   }
> {code}
> There has historically been a quite a few bugs regarding leaked locks.  The 
> Driver will signal the session {{Thread}} by performing an interrupt.  That 
> interrupt can happen any time and it can kill a create/delete action within 
> the ZK framework.  We can see one example of workaround for this.  If the ZK 
> action is interrupted, simply do it again.  Well, what if it's interrupted 
> yet again?  The lock will be leaked.  Also, when the {{InterruptedException}} 
> is caught in the try block, the thread's interrupted flag is cleared.  The 
> flag is not reset in this code and therefore we lose the fact that this 
> thread has been interrupted.  This flag should be preserved so that other 
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
>   Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
>   } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
>   String name = ((ZooKeeperHiveLock)hiveLock).getPath();
>   throw new LockException("Node " + name + " can not be deleted after 
> " + numRetriesForUnLock + " attempts.",
>   e);
> }
>   }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete 
> the lock (again, for fear of leaking it).  This sleep should be 
> uninterruptible.  If we need to get the lock deleted, and there's a problem, 
> interrupting the sleep will cause the code to eventually exit and locks will 
> be leaked.
> It also requires a bunch more TLC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21034) Add option to schematool to drop Hive databases

2019-03-19 Thread Daniel Voros (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796118#comment-16796118
 ] 

Daniel Voros commented on HIVE-21034:
-

Attached patch #5, that removes the getter+setter methods and adds 
{{@VisibleForTesting}} to the {{yes}} field.

> Add option to schematool to drop Hive databases
> ---
>
> Key: HIVE-21034
> URL: https://issues.apache.org/jira/browse/HIVE-21034
> Project: Hive
>  Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21034.1.patch, HIVE-21034.2.patch, 
> HIVE-21034.2.patch, HIVE-21034.3.patch, HIVE-21034.4.patch, HIVE-21034.5.patch
>
>
> An option to remove all Hive managed data could be a useful addition to 
> {{schematool}}.
> I propose to introduce a new flag {{-dropAllDatabases}} that would *drop all 
> databases with CASCADE* to remove all data of managed tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21034) Add option to schematool to drop Hive databases

2019-03-19 Thread Daniel Voros (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Voros updated HIVE-21034:

Attachment: HIVE-21034.5.patch

> Add option to schematool to drop Hive databases
> ---
>
> Key: HIVE-21034
> URL: https://issues.apache.org/jira/browse/HIVE-21034
> Project: Hive
>  Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21034.1.patch, HIVE-21034.2.patch, 
> HIVE-21034.2.patch, HIVE-21034.3.patch, HIVE-21034.4.patch, HIVE-21034.5.patch
>
>
> An option to remove all Hive managed data could be a useful addition to 
> {{schematool}}.
> I propose to introduce a new flag {{-dropAllDatabases}} that would *drop all 
> databases with CASCADE* to remove all data of managed tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21460) ACID: Load data followed by a select * query results in incorrect results

2019-03-19 Thread Victor Corral (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796113#comment-16796113
 ] 

Victor Corral commented on HIVE-21460:
--

http://mail-archives.apache.org/mod_mbox/infra-issues/201903.mbox/raw/.com]

> ACID: Load data followed by a select * query results in incorrect results
> -
>
> Key: HIVE-21460
> URL: https://issues.apache.org/jira/browse/HIVE-21460
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Blocker
> Attachments: HIVE-21460.1.patch
>
>
> This affects current master as well. Created an orc file such that it spans 
> multiple stripes and ran a simple select *, and got incorrect row counts 
> (when comparing with select count(*). The problem seems to be that after 
> split generation and creating min/max rowId for each row (note that since the 
> loaded file is not written by Hive ACID, it does not have ROW__ID in the 
> file; but the ROW__ID is applied on read by discovering min/max bounds which 
> are used for calculating ROW__ID.rowId for each row of a split), Hive is only 
> reading the last split.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21460) ACID: Load data followed by a select * query results in incorrect results

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796112#comment-16796112
 ] 

Hive QA commented on HIVE-21460:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962943/HIVE-21460.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15833 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce] 
(batchId=61)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
 (batchId=99)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16571/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16571/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16571/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962943 - PreCommit-HIVE-Build

> ACID: Load data followed by a select * query results in incorrect results
> -
>
> Key: HIVE-21460
> URL: https://issues.apache.org/jira/browse/HIVE-21460
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Blocker
> Attachments: HIVE-21460.1.patch
>
>
> This affects current master as well. Created an orc file such that it spans 
> multiple stripes and ran a simple select *, and got incorrect row counts 
> (when comparing with select count(*). The problem seems to be that after 
> split generation and creating min/max rowId for each row (note that since the 
> loaded file is not written by Hive ACID, it does not have ROW__ID in the 
> file; but the ROW__ID is applied on read by discovering min/max bounds which 
> are used for calculating ROW__ID.rowId for each row of a split), Hive is only 
> reading the last split.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21474) Bumping guava version

2019-03-19 Thread Peter Vary (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-21474:
--
Status: Patch Available  (was: Open)

> Bumping guava version
> -
>
> Key: HIVE-21474
> URL: https://issues.apache.org/jira/browse/HIVE-21474
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21474.patch
>
>
> Bump guava to 24.1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21474) Bumping guava version

2019-03-19 Thread Peter Vary (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796076#comment-16796076
 ] 

Peter Vary commented on HIVE-21474:
---

[~kgyrtkirk]: The only somewhat meaningful change is in BasicStats class. Could 
you please review?

Thanks,

Peter

> Bumping guava version
> -
>
> Key: HIVE-21474
> URL: https://issues.apache.org/jira/browse/HIVE-21474
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21474.patch
>
>
> Bump guava to 24.1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21474) Bumping guava version

2019-03-19 Thread Peter Vary (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-21474:
--
Attachment: HIVE-21474.patch

> Bumping guava version
> -
>
> Key: HIVE-21474
> URL: https://issues.apache.org/jira/browse/HIVE-21474
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21474.patch
>
>
> Bump guava to 24.1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21473) Bumping jackson version

2019-03-19 Thread Marta Kuczora (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796065#comment-16796065
 ] 

Marta Kuczora commented on HIVE-21473:
--

+1 pending tests

> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21473.patch
>
>
> Bump jackson version to 2.9.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21460) ACID: Load data followed by a select * query results in incorrect results

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796069#comment-16796069
 ] 

Hive QA commented on HIVE-21460:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
15s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 22s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16571/dev-support/hive-personality.sh
 |
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16571/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID: Load data followed by a select * query results in incorrect results
> -
>
> Key: HIVE-21460
> URL: https://issues.apache.org/jira/browse/HIVE-21460
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Blocker
> Attachments: HIVE-21460.1.patch
>
>
> This affects current master as well. Created an orc file such that it spans 
> multiple stripes and ran a simple select *, and got incorrect row counts 
> (when comparing with select count(*). The problem seems to be that after 
> split generation and creating min/max rowId for each row (note that since the 
> loaded file is not written by Hive ACID, it does not have ROW__ID in the 
> file; but the ROW__ID is applied on read by discovering min/max bounds which 
> are used for calculating ROW__ID.rowId for each row of a split), Hive is only 
> reading the last split.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21473) Bumping jackson version

2019-03-19 Thread Peter Vary (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796053#comment-16796053
 ] 

Peter Vary commented on HIVE-21473:
---

[~kuczoram]: Could you please review?

Thanks,

Peter

> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21473.patch
>
>
> Bump jackson version to 2.9.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-21474) Bumping guava version

2019-03-19 Thread Peter Vary (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary reassigned HIVE-21474:
-


> Bumping guava version
> -
>
> Key: HIVE-21474
> URL: https://issues.apache.org/jira/browse/HIVE-21474
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
>
> Bump guava to 24.1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21473) Bumping jackson version

2019-03-19 Thread Peter Vary (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-21473:
--
Status: Patch Available  (was: Open)

> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21473.patch
>
>
> Bump jackson version to 2.9.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21473) Bumping jackson version

2019-03-19 Thread Peter Vary (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-21473:
--
Attachment: HIVE-21473.patch

> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21473.patch
>
>
> Bump jackson version to 2.9.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-21473) Bumping jackson version

2019-03-19 Thread Peter Vary (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary reassigned HIVE-21473:
-


> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
>  Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
>
> Bump jackson version to 2.9.8



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796040#comment-16796040
 ] 

Hive QA commented on HIVE-21430:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962926/HIVE-21430.03.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15834 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
 (batchId=99)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16570/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16570/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16570/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962926 - PreCommit-HIVE-Build

> INSERT into a dynamically partitioned table with hive.stats.autogather = 
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch, 
> HIVE-21430.03.patch, metaexception_repro.patch, 
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
>   Original Estimate: 48h
>  Time Spent: 50m
>  Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the 
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state 
> for a transactional table default.simple_stats without providing the 
> transactional write state for verification (new write ID 5, valid write IDs 
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new 
> state null
>  at 
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796008#comment-16796008
 ] 

Hive QA commented on HIVE-21430:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
8s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 17s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16570/dev-support/hive-personality.sh
 |
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16570/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> INSERT into a dynamically partitioned table with hive.stats.autogather = 
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch, 
> HIVE-21430.03.patch, metaexception_repro.patch, 
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
>   Original Estimate: 48h
>  Time Spent: 50m
>  Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the 
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state 
> for a transactional table default.simple_stats without providing the 
> transactional write state for verification (new write ID 5, valid write IDs 
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new 
> state null
>  at 
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications

2019-03-19 Thread Sankar Hariappan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795990#comment-16795990
 ] 

Sankar Hariappan commented on HIVE-21446:
-

+1 for 03.patch, pending tests

> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, 
> HIVE-21446.03.patch
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21456) Hive Metastore HTTP Thrift

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795987#comment-16795987
 ] 

Hive QA commented on HIVE-21456:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962912/HIVE-21456.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
 (batchId=99)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16569/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16569/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16569/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12962912 - PreCommit-HIVE-Build

> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Amit Khanna
>Priority: Major
> Attachments: HIVE-21456.2.patch, HIVE-21456.3.patch, HIVE-21456.patch
>
>
> Hive Metastore currently doesn't have support for HTTP transport because of 
> which it is not possible to access it via Knox. Adding support for Thrift 
> over HTTP transport will allow the clients to access via Knox



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-21456) Hive Metastore HTTP Thrift

2019-03-19 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795962#comment-16795962
 ] 

Hive QA commented on HIVE-21456:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
30s{color} | {color:blue} standalone-metastore/metastore-common in master has 
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m 
17s{color} | {color:blue} standalone-metastore/metastore-server in master has 
179 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16569/dev-support/hive-personality.sh
 |
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16569/yetus/whitespace-eol.txt
 |
| modules | C: standalone-metastore/metastore-common 
standalone-metastore/metastore-server U: standalone-metastore |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16569/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Amit Khanna
>Priority: Major
> Attachments: HIVE-21456.2.patch, HIVE-21456.3.patch, HIVE-21456.patch
>
>
> Hive Metastore currently doesn't have support for HTTP transport because of 
> which it is not possible to access it via Knox. Adding support for Thrift 
> over HTTP transport will allow the clients to access via Knox



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21283) Create Synonym mid for substr, position for locate

2019-03-19 Thread Mani M (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mani M updated HIVE-21283:
--
Status: Patch Available  (was: In Progress)

Resubmitting the same patch again to avoid the failure due to falky test cases

> Create Synonym mid for  substr, position for  locate
> 
>
> Key: HIVE-21283
> URL: https://issues.apache.org/jira/browse/HIVE-21283
> Project: Hive
>  Issue Type: New Feature
>Reporter: Mani M
>Assignee: Mani M
>Priority: Minor
>  Labels: UDF, pull-request-available, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, 
> HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, 
> HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.2.PATCH, 
> HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, 
> image-2019-03-16-21-33-18-898.png
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Create new synonym for the existing function
>  
> Mid for substr
> postiion for locate 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-21290) Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time

2019-03-19 Thread Karen Coppage (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karen Coppage reassigned HIVE-21290:


Assignee: Karen Coppage

> Restore historical way of handling timestamps in Parquet while keeping the 
> new semantics at the same time
> -
>
> Key: HIVE-21290
> URL: https://issues.apache.org/jira/browse/HIVE-21290
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Ivanfi
>Assignee: Karen Coppage
>Priority: Major
>
> This sub-task is for implementing the Parquet-specific parts of the following 
> plan:
> h1. Problem
> Historically, the semantics of the TIMESTAMP type in Hive depended on the 
> file format. Timestamps in Avro, Parquet and RCFiles with a binary SerDe had 
> _Instant_ semantics, while timestamps in ORC, textfiles and RCFiles with a 
> text SerDe had _LocalDateTime_ semantics.
> The Hive community wanted to get rid of this inconsistency and have 
> _LocalDateTime_ semantics in Avro, Parquet and RCFiles with a binary SerDe as 
> well. *Hive 3.1 turned off normalization to UTC* to achieve this. While this 
> leads to the desired new semantics, it also leads to incorrect results when 
> new Hive versions read timestamps written by old Hive versions or when old 
> Hive versions or any other component not aware of this change (including 
> legacy Impala and Spark versions) read timestamps written by new Hive 
> versions.
> h1. Solution
> To work around this issue, Hive *should restore the practice of normalizing 
> to UTC* when writing timestamps to Avro, Parquet and RCFiles with a binary 
> SerDe. In itself, this would restore the historical _Instant_ semantics, 
> which is undesirable. In order to achieve the desired _LocalDateTime_ 
> semantics in spite of normalizing to UTC, newer Hive versions should record 
> the session-local local time zone in the file metadata fields serving 
> arbitrary key-value storage purposes.
> When reading back files with this time zone metadata, newer Hive versions (or 
> any other new component aware of this extra metadata) can achieve 
> _LocalDateTime_ semantics by *converting from UTC to the saved time zone 
> (instead of to the local time zone)*. Legacy components that are unaware of 
> the new metadata can read the files without any problem and the timestamps 
> will show the historical Instant behaviour to them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan

2019-03-19 Thread Zoltan Haindrich (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-21304:

Attachment: HIVE-21304.01.patch

> Show Bucketing version for ReduceSinkOp in explain extended plan
> 
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>  
> cc [~vgarg]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-20934) ACID: Query based compactor for minor compaction

2019-03-19 Thread Aron Hamvas (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aron Hamvas reassigned HIVE-20934:
--

Assignee: Aron Hamvas  (was: Vaibhav Gumashta)

> ACID: Query based compactor for minor compaction
> 
>
> Key: HIVE-20934
> URL: https://issues.apache.org/jira/browse/HIVE-20934
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Aron Hamvas
>Priority: Major
>
> Follow up of HIVE-20699. This is to enable running minor compactions as a 
> HiveQL query 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-13479) Relax sorting requirement in ACID tables

2019-03-19 Thread Abhishek Somani (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795933#comment-16795933
 ] 

Abhishek Somani commented on HIVE-13479:


What I meant is now that ACID v2 has been implemented, do we plan to work on 
relaxing the sorting requirement? As far as I know, we still enforce that the 
rows be sorted on the acid columns(row id), and this is done so that the reader 
can sort-merge the delete events with the insert events while reading. Isn't 
that right?

If so, it seems the only way to have data sorted on another column specified by 
the user seems to be to initially insert the data with ordering on that column, 
so that the data is sorted BOTH on the acid columns as well as user specified 
column.

If however we were able to relax the requirement that data HAS to be sorted on 
the acid columns, we could utilize something like compaction to sort the data 
on user desired columns in the background. Theoretically one could do such 
sorting in compaction even today, but if the sorting requirement is not 
relaxed, we will need to sort both on row ids and user-column, for which one 
would need the compaction to behave as an insert overwrite and generate new row 
ids so that the data is sorted on both the (new)row id columns as well as the 
user specified column, which would be good to avoid.

Have I understood this correct?

> Relax sorting requirement in ACID tables
> 
>
> Key: HIVE-13479
> URL: https://issues.apache.org/jira/browse/HIVE-13479
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 1.2.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>   Original Estimate: 160h
>  Remaining Estimate: 160h
>
> Currently ACID tables require data to be sorted according to internal primary 
> key.  This is that base + delta files can be efficiently sort/merged to 
> produce the snapshot for current transaction.
> This prevents the user to make the table sorted based on any other criteria 
> which can be useful.  One example is using dynamic partition insert (which 
> also occurs for update/delete SQL).  This may create lots of writers 
> (buckets*partitions) and tax cluster resources.
> The usual solution is hive.optimize.sort.dynamic.partition=true which won't 
> be honored for ACID tables.
> We could rely on hash table based algorithm to merge delta files and then not 
> require any particular sort on Acid tables.  One way to do that is to treat 
> each update event as an Insert (new internal PK) + delete (old PK).  Delete 
> events are very small since they just need to contain PKs.  So the hash table 
> would just need to contain Delete events and be reasonably memory efficient.
> This is a significant amount of work but worth doing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-20580) OrcInputFormat.isOriginal() should not rely on hive.acid.key.index

2019-03-19 Thread Peter Vary (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795934#comment-16795934
 ] 

Peter Vary commented on HIVE-20580:
---

[~ekoifman]: The {{isOriginal(Footer footer)}} is called from 
{{org.apache.hadoop.hive.llap.io.metadata.OrcFileMetadata}} constructor, and 
the value set by the constructor is only used to implement the 
{{org.apache.orc.FileMetadata.isOriginalFormat()}} interface method which in 
turn is not called anywhere in the Hive code base.
 Since this is an external interface I rather not leave the method 
unimplemented, even if it is not called anywhere at the moment. On the other 
hand you confirmed that I correctly understood the meaning of the isOriginal, 
so I think it is ok to fix the output of the other method to match the same as 
well.

Thanks for the help [~ekoifman]!!!
 Really appreciate it!

Peter

> OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
> --
>
> Key: HIVE-20580
> URL: https://issues.apache.org/jira/browse/HIVE-20580
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20580.2.patch, HIVE-20580.3.patch, 
> HIVE-20580.4.patch, HIVE-20580.5.patch, HIVE-20580.6.patch, HIVE-20580.patch
>
>
> {{org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isOriginal()}} is checking 
> for presence of {{hive.acid.key.index}} in the footer.  This is only created 
> when the file is written by {{OrcRecordUpdater}}.  It should instead check 
> for presence of Acid metadata columns so that a file can be produced by 
> something other than {{OrcRecordUpater}}.
> Also, {{hive.acid.key.index}} counts number of different type of events which 
> is not really useful for Acid V2 (as of Hive 3) since each file only has 1 
> type of event.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-21283) Create Synonym mid for substr, position for locate

2019-03-19 Thread Mani M (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mani M updated HIVE-21283:
--
Attachment: HIVE.21283.09.PATCH

> Create Synonym mid for  substr, position for  locate
> 
>
> Key: HIVE-21283
> URL: https://issues.apache.org/jira/browse/HIVE-21283
> Project: Hive
>  Issue Type: New Feature
>Reporter: Mani M
>Assignee: Mani M
>Priority: Minor
>  Labels: UDF, pull-request-available, todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, 
> HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, 
> HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.2.PATCH, 
> HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, 
> image-2019-03-16-21-33-18-898.png
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Create new synonym for the existing function
>  
> Mid for substr
> postiion for locate 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


  1   2   >