[jira] [Commented] (HIVE-19269) Vectorization: Turn On by Default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455797#comment-16455797
 ] 

Hive QA commented on HIVE-19269:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920838/HIVE-19269.05.patch

{color:green}SUCCESS:{color} +1 due to 154 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 61 failed/errored test(s), 14284 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[reduce_deduplicate_extended]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[retry_failure]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[retry_failure_oom]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[retry_failure_stat_changes]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing] 
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[windowing_gby]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[bucket_mapjoin_mismatch1]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
 (batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg] 
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error] 
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
 (batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
 (batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
 (batchId=231)
org.apache.hadoop.hive.ql.plan.mapping.TestOperatorCmp.testDifferentFiltersAreNotMatched
 (batchId=298)
org.apache.hadoop.hive.ql.plan.mapping.TestOperatorCmp.testSameFiltersMatched 
(batchId=298)
org.apache.hadoop.hive.ql.plan.mapping.TestOperatorCmp.testUnrelatedFiltersAreNotMatched0
 (batchId=298)
org.apache.hadoop.hive.ql.plan.mapping.TestOperatorCmp.testUnrelatedFiltersAreNotMatched1
 (batchId=298)
org.apache.hadoop.hive.ql.plan.mapping.TestReOptimization.testNotReExecutedIfAssertionError
 (batchId=298)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=240)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 
(batchId=242)

[jira] [Commented] (HIVE-19269) Vectorization: Turn On by Default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455779#comment-16455779
 ] 

Hive QA commented on HIVE-19269:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  3m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10519/dev-support/hive-personality.sh
 |
| git revision | master / 0dec595 |
| Default Java | 1.8.0_111 |
| modules | C: common accumulo-handler contrib hbase-handler 
itests/hive-blobstore itests/hive-unit ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10519/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Vectorization: Turn On by Default
> -
>
> Key: HIVE-19269
> URL: https://issues.apache.org/jira/browse/HIVE-19269
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19269.01.patch, HIVE-19269.02.patch, 
> HIVE-19269.04.patch, HIVE-19269.05.patch
>
>
> Reflect that our most expected Hive deployment will be using vectorization 
> and change the default of hive.vectorized.execution.enabled to true.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19110) Vectorization: Enabling vectorization causes TestContribCliDriver udf_example_arraymapstruct.q to produce Wrong Results

[Haifeng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haifeng Chen updated HIVE-19110:

Status: Patch Available  (was: Open)

New patch based on the latest source code. With the latest source code, #1 and 
#2 issues mentioned above were fixed. #3 is fixed in this patch and unit tests 
and one qtest for testing list and map vector expression writers are added. 

> Vectorization: Enabling vectorization causes TestContribCliDriver 
> udf_example_arraymapstruct.q to produce Wrong Results
> ---
>
> Key: HIVE-19110
> URL: https://issues.apache.org/jira/browse/HIVE-19110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Haifeng Chen
>Priority: Critical
> Attachments: HIVE-19110.01.patch, HIVE-19110.02.patch
>
>
> Found in vectorization enable by default experiment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19110) Vectorization: Enabling vectorization causes TestContribCliDriver udf_example_arraymapstruct.q to produce Wrong Results

[Haifeng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haifeng Chen updated HIVE-19110:

Status: Open  (was: Patch Available)

> Vectorization: Enabling vectorization causes TestContribCliDriver 
> udf_example_arraymapstruct.q to produce Wrong Results
> ---
>
> Key: HIVE-19110
> URL: https://issues.apache.org/jira/browse/HIVE-19110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Haifeng Chen
>Priority: Critical
> Attachments: HIVE-19110.01.patch, HIVE-19110.02.patch
>
>
> Found in vectorization enable by default experiment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19110) Vectorization: Enabling vectorization causes TestContribCliDriver udf_example_arraymapstruct.q to produce Wrong Results

[Haifeng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haifeng Chen updated HIVE-19110:

Attachment: HIVE-19110.02.patch

> Vectorization: Enabling vectorization causes TestContribCliDriver 
> udf_example_arraymapstruct.q to produce Wrong Results
> ---
>
> Key: HIVE-19110
> URL: https://issues.apache.org/jira/browse/HIVE-19110
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Haifeng Chen
>Priority: Critical
> Attachments: HIVE-19110.01.patch, HIVE-19110.02.patch
>
>
> Found in vectorization enable by default experiment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19239) Check for possible null timestamp fields during SerDe from Druid events

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455756#comment-16455756
 ] 

Ashutosh Chauhan commented on HIVE-19239:
-

[~bslim] If this patch is still relevant can you answer my question?

> Check for possible null timestamp fields during SerDe from Druid events
> ---
>
> Key: HIVE-19239
> URL: https://issues.apache.org/jira/browse/HIVE-19239
> Project: Hive
>  Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19239.patch
>
>
> Currently we do not check for possible null timestamp events.
> This might lead to NPE.
> This Patch add addition check for such case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19294) grouping sets when contains a constant column

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455755#comment-16455755
 ] 

Ashutosh Chauhan commented on HIVE-19294:
-

Sounds like a bug in Hive's constant folding rule. a can either be 'all' or 
null after grouping set inner query so outer query case statement folding to  
CASE WHEN (true) THEN ('x') is incorrect.

> grouping sets when contains a constant column
> -
>
> Key: HIVE-19294
> URL: https://issues.apache.org/jira/browse/HIVE-19294
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 2.3.2
>Reporter: Song Jun
>Priority: Major
>
> We have different results between Hive-1.2.2 and Hive-2.3.2, SQL like this:
> {code:java}
> select 
> case when a='all' then 'x' 
>  when b=1 then 'y' 
>  else 'z' 
> end, c 
> from ( 
> select 
>  a,b,count(1) as c
> from ( 
>  select 
> 'all' as a,b 
>  from test 
> ) t1 group by a,b grouping sets(a,b) 
> ) t2;
> {code}
> We have a grouping sets using the column a which is a contant value 'all' in 
> its subquery.
>  
> The result of Hive 1.2.2(same result when set hive.cbo.enable to true or 
> false):
> {code:java}
> x 3
> y 2
> z 1   {code}
> The result of Hive 2.3.2(same result when set hive.cbo.enable to true or 
> false):
> {code:java}
> x 3
> x 2
> x 1{code}
> I dig it out for Hive 2.3.2 and set hive.cbo.enable=false, I found it that 
> the optimizer 
> ConstantPropagate optimize the code according to the constant column value 
> 'all' in the subquery:
> {code:java}
> case when a='all' then 'x' 
>  when b=1 then 'y' 
>  else 'z' 
> end
> {code}
> to 
> {code:java}
> Select Operator
> expressions: CASE WHEN (true) THEN ('x') WHEN ((_col1 = 1)) THEN ('y') ELSE 
> ('z') END (type: string), _col3 (type: bigint)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 3 Data size: 3 Basic stats: COMPLETE Column stats: NONE
> {code}
> That is case when a = 'all'  explained as case when (true), so we always has 
> the value of 'x'.
>  
> So, which should be right for the above query case? 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19306) Arrow batch serializer

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455747#comment-16455747
 ] 

Hive QA commented on HIVE-19306:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920830/HIVE-19306.2.patch

{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 14320 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
 (batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
 (batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
 (batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
 (batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth 
(batchId=254)
{noformat}

Test results: 

[jira] [Commented] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455740#comment-16455740
 ] 

ASF GitHub Bot commented on HIVE-19340:
---

GitHub user maheshk114 opened a pull request:

https://github.com/apache/hive/pull/337

HIVE-19340 : Disable timeout of transactions opened by replication ta…


The transactions opened by applying EVENT_OPEN_TXN should never be aborted 
automatically due to time-out. Aborting of transaction started by replication 
task may leads to inconsistent state at target which needs additional overhead 
to clean-up. So, it is proposed to mark the transactions opened by replication 
task as special ones and shouldn't be aborted if heart beat is lost. This helps 
to ensure all ABORT and COMMIT events will always find the corresponding txn at 
target to operate.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maheshk114/hive BUG-92700

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/337.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #337


commit 317d29c8455ad8aaccf1689c66d79f7bab41cde7
Author: Mahesh Kumar Behera 
Date:   2018-04-27T03:24:08Z

HIVE-19340 : Disable timeout of transactions opened by replication task at 
target cluster




> Disable timeout of transactions opened by replication task at target cluster
> 
>
> Key: HIVE-19340
> URL: https://issues.apache.org/jira/browse/HIVE-19340
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-19340.01.patch
>
>
> The transactions opened by applying EVENT_OPEN_TXN should never be aborted 
> automatically due to time-out. Aborting of transaction started by replication 
> task may leads to inconsistent state at target which needs additional 
> overhead to clean-up. So, it is proposed to mark the transactions opened by 
> replication task as special ones and shouldn't be aborted if heart beat is 
> lost. This helps to ensure all ABORT and COMMIT events will always find the 
> corresponding txn at target to operate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-19340:
---
Status: Patch Available  (was: Open)

> Disable timeout of transactions opened by replication task at target cluster
> 
>
> Key: HIVE-19340
> URL: https://issues.apache.org/jira/browse/HIVE-19340
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-19340.01.patch
>
>
> The transactions opened by applying EVENT_OPEN_TXN should never be aborted 
> automatically due to time-out. Aborting of transaction started by replication 
> task may leads to inconsistent state at target which needs additional 
> overhead to clean-up. So, it is proposed to mark the transactions opened by 
> replication task as special ones and shouldn't be aborted if heart beat is 
> lost. This helps to ensure all ABORT and COMMIT events will always find the 
> corresponding txn at target to operate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-19340:
---
Attachment: HIVE-19340.01.patch

> Disable timeout of transactions opened by replication task at target cluster
> 
>
> Key: HIVE-19340
> URL: https://issues.apache.org/jira/browse/HIVE-19340
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-19340.01.patch
>
>
> The transactions opened by applying EVENT_OPEN_TXN should never be aborted 
> automatically due to time-out. Aborting of transaction started by replication 
> task may leads to inconsistent state at target which needs additional 
> overhead to clean-up. So, it is proposed to mark the transactions opened by 
> replication task as special ones and shouldn't be aborted if heart beat is 
> lost. This helps to ensure all ABORT and COMMIT events will always find the 
> corresponding txn at target to operate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19306) Arrow batch serializer

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455736#comment-16455736
 ] 

Hive QA commented on HIVE-19306:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m  
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m  
1s{color} | {color:red} root: The patch generated 655 new + 427 unchanged - 0 
fixed = 1082 total (was 427) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
47s{color} | {color:red} ql: The patch generated 655 new + 0 unchanged - 0 
fixed = 655 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  6m 
15s{color} | {color:red} root generated 1 new + 333 unchanged - 1 fixed = 334 
total (was 334) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m  
2s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100 total 
(was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 20s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10518/dev-support/hive-personality.sh
 |
| git revision | master / 0dec595 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10518/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10518/yetus/diff-checkstyle-ql.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10518/yetus/diff-javadoc-javadoc-root.txt
 |
| javadoc | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10518/yetus/diff-javadoc-javadoc-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10518/yetus/patch-asflicense-problems.txt
 |
| modules | C: common serde . ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10518/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Arrow batch serializer
> --
>
> Key: HIVE-19306
> URL: https://issues.apache.org/jira/browse/HIVE-19306
> Project: Hive
>  Issue Type: Task
>  Components: Serializers/Deserializers
>Reporter: Eric Wohlstadter
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-19306.2.patch
>
>
> Leverage the 

[jira] [Commented] (HIVE-19311) Partition and bucketing support for “load data” statement

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455693#comment-16455693
 ] 

Hive QA commented on HIVE-19311:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920729/HIVE-19311.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 232 failed/errored test(s), 14285 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like] 
(batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
 (batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
 (batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
 (batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
 (batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
 (batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
 (batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
 (batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
 (batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join] 
(batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[load_data] 
(batchId=256)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_rename_table] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_cascade] 
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_stats_status]
 (batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1] 
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2] 
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3] 
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_1] 
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_2] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_3] 
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_4] 
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_6] 
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_7] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_8] 
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketizedhiveinputformat_auto]
 (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin10] 
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin11] 
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12] 
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5] 
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin8] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative2] 
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative] 
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl] 
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl_dp] 
(batchId=53)

[jira] [Updated] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera updated HIVE-19340:
---
Description: The transactions opened by applying EVENT_OPEN_TXN should 
never be aborted automatically due to time-out. Aborting of transaction started 
by replication task may leads to inconsistent state at target which needs 
additional overhead to clean-up. So, it is proposed to mark the transactions 
opened by replication task as special ones and shouldn't be aborted if heart 
beat is lost. This helps to ensure all ABORT and COMMIT events will always find 
the corresponding txn at target to operate.  (was:  
h1. Replicate ACID write Events
 * Create new EVENT_WRITE event with related message format to log the write 
operations with in a txn along with data associated.

 * Log this event when perform any writes (insert into, insert overwrite, load 
table, delete, update, merge, truncate) on table/partition.

 * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple 
partitions, then need to log one event per partition.

 * DbNotificationListener should log this type of event to special metastore 
table named "MTxnWriteNotificationLog".

 * This table should maintain a map of txn ID against list of tables/partitions 
written by given txn.

 * The entry for a given txn should be removed by the cleaner thread that 
removes the expired events from EventNotificationTable.

h1. Replicate Commit Txn operation (with writes)

Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions 
modified within the txn.

*Source warehouse:*
 * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" 
metastore table to consolidate the list of tables/partitions modified within 
this txn scope.

 * Based on the list of tables/partitions modified and table Write ID, need to 
compute the list of delta files added by this txn.

 * Repl dump should read this message and dump the metadata and delta files 
list.

*Target warehouse:*
 * Ensure snapshot isolation at target for on-going read txns which shouldn't 
view the data replicated from committed txn. (Ensured with open and allocate 
write ID events).)

> Disable timeout of transactions opened by replication task at target cluster
> 
>
> Key: HIVE-19340
> URL: https://issues.apache.org/jira/browse/HIVE-19340
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
>
> The transactions opened by applying EVENT_OPEN_TXN should never be aborted 
> automatically due to time-out. Aborting of transaction started by replication 
> task may leads to inconsistent state at target which needs additional 
> overhead to clean-up. So, it is proposed to mark the transactions opened by 
> replication task as special ones and shouldn't be aborted if heart beat is 
> lost. This helps to ensure all ABORT and COMMIT events will always find the 
> corresponding txn at target to operate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster

[mahesh kumar behera (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mahesh kumar behera reassigned HIVE-19340:
--


> Disable timeout of transactions opened by replication task at target cluster
> 
>
> Key: HIVE-19340
> URL: https://issues.apache.org/jira/browse/HIVE-19340
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
>  Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
>
>  
> h1. Replicate ACID write Events
>  * Create new EVENT_WRITE event with related message format to log the write 
> operations with in a txn along with data associated.
>  * Log this event when perform any writes (insert into, insert overwrite, 
> load table, delete, update, merge, truncate) on table/partition.
>  * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple 
> partitions, then need to log one event per partition.
>  * DbNotificationListener should log this type of event to special metastore 
> table named "MTxnWriteNotificationLog".
>  * This table should maintain a map of txn ID against list of 
> tables/partitions written by given txn.
>  * The entry for a given txn should be removed by the cleaner thread that 
> removes the expired events from EventNotificationTable.
> h1. Replicate Commit Txn operation (with writes)
> Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions 
> modified within the txn.
> *Source warehouse:*
>  * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" 
> metastore table to consolidate the list of tables/partitions modified within 
> this txn scope.
>  * Based on the list of tables/partitions modified and table Write ID, need 
> to compute the list of delta files added by this txn.
>  * Repl dump should read this message and dump the metadata and delta files 
> list.
> *Target warehouse:*
>  * Ensure snapshot isolation at target for on-going read txns which shouldn't 
> view the data replicated from committed txn. (Ensured with open and allocate 
> write ID events).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19311) Partition and bucketing support for “load data” statement

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455689#comment-16455689
 ] 

Hive QA commented on HIVE-19311:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  2m  
6s{color} | {color:red} root: The patch generated 13 new + 734 unchanged - 14 
fixed = 747 total (was 748) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
48s{color} | {color:red} ql: The patch generated 13 new + 734 unchanged - 14 
fixed = 747 total (was 748) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 20s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10517/dev-support/hive-personality.sh
 |
| git revision | master / 0dec595 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10517/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10517/yetus/diff-checkstyle-ql.txt
 |
| modules | C: . ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10517/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Partition and bucketing support for “load data” statement
> -
>
> Key: HIVE-19311
> URL: https://issues.apache.org/jira/browse/HIVE-19311
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-19311.1.patch, HIVE-19311.2.patch
>
>
> Currently, "load data" statement is very limited. It errors out if any of the 
> information is missing such as partitioning info if table is partitioned or 
> appropriate names when table is bucketed.
> It should be able to launch an insert job to load the data instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455678#comment-16455678
 ] 

Vineet Garg commented on HIVE-19322:


I ran the tests on branch-3

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455676#comment-16455676
 ] 

Vineet Garg commented on HIVE-19322:


[~ekoifman] I just ran {{subquery_in_implicit_gby}} without HIVE-18739 and then 
with HIVE-18739. It passes without HIVE-18739 and fails with the patch with 
{{Error during job, obtaining debugging information...}}.
I am certain HIVE-18739 caused all of these failures.

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-6980) Drop table by using direct sql

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455639#comment-16455639
 ] 

Hive QA commented on HIVE-6980:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920826/HIVE-6980.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 14280 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
 (batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
 (batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
 (batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
 (batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth 

[jira] [Updated] (HIVE-19339) Regenerate alltypesorc file with latest ORC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-19339:
---
Attachment: (was: HIVE-19339.patch)

> Regenerate alltypesorc file with latest ORC
> ---
>
> Key: HIVE-19339
> URL: https://issues.apache.org/jira/browse/HIVE-19339
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19339.patch
>
>
> Among others, new files contain timezone information in the stripe footer. We 
> want to run tests over {{alltypesorc}} file generated using more recent 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19339) Regenerate alltypesorc file with latest ORC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-19339:
---
Attachment: HIVE-19339.patch

> Regenerate alltypesorc file with latest ORC
> ---
>
> Key: HIVE-19339
> URL: https://issues.apache.org/jira/browse/HIVE-19339
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19339.patch
>
>
> Among others, new files contain timezone information in the stripe footer. We 
> want to run tests over {{alltypesorc}} file generated using more recent 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19339) Regenerate alltypesorc file with latest ORC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-19339:
---
Attachment: HIVE-19339.patch

> Regenerate alltypesorc file with latest ORC
> ---
>
> Key: HIVE-19339
> URL: https://issues.apache.org/jira/browse/HIVE-19339
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19339.patch
>
>
> Among others, new files contain timezone information in the stripe footer. We 
> want to run tests over {{alltypesorc}} file generated using more recent 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19339) Regenerate alltypesorc file with latest ORC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-19339:
--


> Regenerate alltypesorc file with latest ORC
> ---
>
> Key: HIVE-19339
> URL: https://issues.apache.org/jira/browse/HIVE-19339
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Among others, new files contain timezone information in the stripe footer. We 
> want to run tests over {{alltypesorc}} file generated using more recent 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19339) Regenerate alltypesorc file with latest ORC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-19339:
---
Status: Patch Available  (was: In Progress)

> Regenerate alltypesorc file with latest ORC
> ---
>
> Key: HIVE-19339
> URL: https://issues.apache.org/jira/browse/HIVE-19339
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Among others, new files contain timezone information in the stripe footer. We 
> want to run tests over {{alltypesorc}} file generated using more recent 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-19339) Regenerate alltypesorc file with latest ORC

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-19339 started by Jesus Camacho Rodriguez.
--
> Regenerate alltypesorc file with latest ORC
> ---
>
> Key: HIVE-19339
> URL: https://issues.apache.org/jira/browse/HIVE-19339
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Among others, new files contain timezone information in the stripe footer. We 
> want to run tests over {{alltypesorc}} file generated using more recent 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19327) qroupby_rollup_empty.q fails for insert-only transactional tables

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455615#comment-16455615
 ] 

Sergey Shelukhin commented on HIVE-19327:
-

The result may be invalid if there are some invalid directories in the table 
(i.e. finalPaths is empty, but there are some aborted deltas, for example). 
This will just add them and read them as a single directory.
See CombinedHiveInputFormat for an example of what it does in this case...
{noformat}
  // If there are no inputs; the Execution engine skips the operator tree.
  // To prevent it from happening; an opaque  ZeroRows input is added here 
- when needed.
  result.add(
  new HiveInputSplit(new NullRowsInputFormat.DummyInputSplit(paths[0]), 
ZeroRowsInputFormat.class.getName()));
{noformat}
That will ensure only one row actually gets produced.

> qroupby_rollup_empty.q fails for insert-only transactional tables
> -
>
> Key: HIVE-19327
> URL: https://issues.apache.org/jira/browse/HIVE-19327
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19327.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19324) improve YARN queue check error message in Tez pool

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455606#comment-16455606
 ] 

Sergey Shelukhin commented on HIVE-19324:
-

I hate HiveQA...

> improve YARN queue check error message in Tez pool
> --
>
> Key: HIVE-19324
> URL: https://issues.apache.org/jira/browse/HIVE-19324
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepesh Khandelwal
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19324.01.patch, HIVE-19324.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19324) improve YARN queue check error message in Tez pool

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19324:

Attachment: HIVE-19324.01.patch

> improve YARN queue check error message in Tez pool
> --
>
> Key: HIVE-19324
> URL: https://issues.apache.org/jira/browse/HIVE-19324
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepesh Khandelwal
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19324.01.patch, HIVE-19324.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19312) MM tables don't work with BucketizedHIF

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455604#comment-16455604
 ] 

Sergey Shelukhin commented on HIVE-19312:
-

I hate HiveQA.

> MM tables don't work with BucketizedHIF
> ---
>
> Key: HIVE-19312
> URL: https://issues.apache.org/jira/browse/HIVE-19312
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19312.01.patch, HIVE-19312.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-6980) Drop table by using direct sql

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455605#comment-16455605
 ] 

Hive QA commented on HIVE-6980:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
27s{color} | {color:red} standalone-metastore: The patch generated 6 new + 675 
unchanged - 5 fixed = 681 total (was 680) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10515/dev-support/hive-personality.sh
 |
| git revision | master / fc42593 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10515/yetus/diff-checkstyle-standalone-metastore.txt
 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10515/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Drop table by using direct sql
> --
>
> Key: HIVE-6980
> URL: https://issues.apache.org/jira/browse/HIVE-6980
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.12.0
>Reporter: Selina Zhang
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-6980.2.patch, HIVE-6980.patch
>
>
> Dropping table which has lots of partitions is slow. Even after applying the 
> patch of HIVE-6265, the drop table still takes hours (100K+ partitions). 
> The fixes come with two parts:
> 1. use directSQL to query the partitions protect mode;
> the current implementation needs to transfer the Partition object to client 
> and check the protect mode for each partition. I'd like to move this part of 
> logic to metastore. The check will be done by direct sql (if direct sql is 
> disabled, execute the same logic in the ObjectStore);
> 2. use directSQL to drop partitions for table;
> there maybe two solutions here:
> 1. add "DELETE CASCADE" in the schema. In this way we only need to delete 
> entries from partitions table use direct sql. May need to change 
> datanucleus.deletionPolicy = DataNucleus. 
> 2. clean up the dependent tables by issue DELETE statement. This also needs 
> to turn on datanucleus.query.sql.allowAll
> Both of above solutions should be able to fix the problem. The DELETE CASCADE 
> has to change schemas and prepare upgrade scripts. The second solutions added 
> maintenance cost if new tables added in the future releases.
> Please advice. 



--
This message was sent 

[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19312:

Attachment: HIVE-19312.01.patch

> MM tables don't work with BucketizedHIF
> ---
>
> Key: HIVE-19312
> URL: https://issues.apache.org/jira/browse/HIVE-19312
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19312.01.patch, HIVE-19312.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19124) implement a basic major compactor for MM tables

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19124:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the reviews!

> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Fix For: 3.0.0
>
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, 
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, 
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, 
> HIVE-19124.08.patch, HIVE-19124.09.patch, HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be 
> supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19317) Handle schema evolution from int like types to decimal

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455588#comment-16455588
 ] 

Hive QA commented on HIVE-19317:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920818/HIVE-19317.1.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 14280 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[typechangetest] 
(batchId=11)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
 (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb] 
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
 (batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
 (batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
 (batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
 (batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.hcatalog.pig.TestParquetHCatStorer.testDateCharTypes 
(batchId=196)
org.apache.hive.hcatalog.pig.TestParquetHCatStorer.testWriteDecimal 
(batchId=196)
org.apache.hive.hcatalog.pig.TestParquetHCatStorer.testWriteDecimalX 
(batchId=196)
org.apache.hive.hcatalog.pig.TestParquetHCatStorer.testWriteDecimalXY 
(batchId=196)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testNonSparkQuery
 (batchId=242)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill 
(batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)

[jira] [Commented] (HIVE-19331) Repl load config in "with" clause not pass to Context.getStagingDir

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455577#comment-16455577
 ] 

Thejas M Nair commented on HIVE-19331:
--

+1 pending tests


> Repl load config in "with" clause not pass to Context.getStagingDir
> ---
>
> Key: HIVE-19331
> URL: https://issues.apache.org/jira/browse/HIVE-19331
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19331.1.patch
>
>
> Another failure similar to HIVE-18626, causing exception when s3 credentials 
> are in "REPL LOAD" with clause.
> {code}
> Caused by: java.lang.IllegalStateException: Error getting FileSystem for 
> s3a://nat-yc-r7-nmys-beacon-cloud-s3-2/hive_incremental_testing.db/hive_incremental_testing_new_tabl...:
>  org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on 
> nat-yc-r7-nmys-beacon-cloud-s3-2: com.amazonaws.AmazonClientException: No AWS 
> Credentials provided by BasicAWSCredentialsProvider 
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
> EC2 metadata service: No AWS Credentials provided by 
> BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
> EC2 metadata service
> at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:359)
> at 
> org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:487)
> at 
> org.apache.hadoop.hive.ql.Context.getExternalTmpPath(Context.java:565)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.loadTable(ImportSemanticAnalyzer.java:370)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.createReplImportTasks(ImportSemanticAnalyzer.java:926)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(ImportSemanticAnalyzer.java:329)
> at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:43)
> ... 24 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19333) Disable condition tree branch removal using stats for external tables

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-19333:
--
Description: Can result in wrong results if branch removal occurs in 
external tables due to out-of-date stats  (was: Can result in wrong results if 
branch removal occurs due to out-of-date stats)

> Disable condition tree branch removal using stats for external tables
> -
>
> Key: HIVE-19333
> URL: https://issues.apache.org/jira/browse/HIVE-19333
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jason Dere
>Priority: Major
>
> Can result in wrong results if branch removal occurs in external tables due 
> to out-of-date stats



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19333) Disable condition tree branch removal using stats for external tables

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-19333:
--
Summary: Disable condition tree branch removal using stats for external 
tables  (was: Disable operator tree branch removal using stats )

> Disable condition tree branch removal using stats for external tables
> -
>
> Key: HIVE-19333
> URL: https://issues.apache.org/jira/browse/HIVE-19333
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jason Dere
>Priority: Major
>
> Can result in wrong results if branch removal occurs due to out-of-date stats



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18910:
--
Attachment: HIVE-18910.44.patch

> Migrate to Murmur hash for shuffle and bucketing
> 
>
> Key: HIVE-18910
> URL: https://issues.apache.org/jira/browse/HIVE-18910
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18910.1.patch, HIVE-18910.10.patch, 
> HIVE-18910.11.patch, HIVE-18910.12.patch, HIVE-18910.13.patch, 
> HIVE-18910.14.patch, HIVE-18910.15.patch, HIVE-18910.16.patch, 
> HIVE-18910.17.patch, HIVE-18910.18.patch, HIVE-18910.19.patch, 
> HIVE-18910.2.patch, HIVE-18910.20.patch, HIVE-18910.21.patch, 
> HIVE-18910.22.patch, HIVE-18910.23.patch, HIVE-18910.24.patch, 
> HIVE-18910.25.patch, HIVE-18910.26.patch, HIVE-18910.27.patch, 
> HIVE-18910.28.patch, HIVE-18910.29.patch, HIVE-18910.3.patch, 
> HIVE-18910.30.patch, HIVE-18910.31.patch, HIVE-18910.32.patch, 
> HIVE-18910.33.patch, HIVE-18910.34.patch, HIVE-18910.35.patch, 
> HIVE-18910.36.patch, HIVE-18910.36.patch, HIVE-18910.37.patch, 
> HIVE-18910.38.patch, HIVE-18910.39.patch, HIVE-18910.4.patch, 
> HIVE-18910.40.patch, HIVE-18910.41.patch, HIVE-18910.42.patch, 
> HIVE-18910.43.patch, HIVE-18910.44.patch, HIVE-18910.5.patch, 
> HIVE-18910.6.patch, HIVE-18910.7.patch, HIVE-18910.8.patch, HIVE-18910.9.patch
>
>
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19323) Create metastore SQL install and upgrade scripts for 3.1

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-19323:
--
Attachment: HIVE-19323.2.patch

> Create metastore SQL install and upgrade scripts for 3.1
> 
>
> Key: HIVE-19323
> URL: https://issues.apache.org/jira/browse/HIVE-19323
> Project: Hive
>  Issue Type: Task
>  Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-19323.2.patch, HIVE-19323.patch
>
>
> Now that we've branched for 3.0 we need to create SQL install and upgrade 
> scripts for 3.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19323) Create metastore SQL install and upgrade scripts for 3.1

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1643#comment-1643
 ] 

Alan Gates commented on HIVE-19323:
---

Second version of the patch fixes an issue where the Hive version name in the 
pom hadn't been updated which was causing the wrong version of the upgrade 
scripts to be run.

> Create metastore SQL install and upgrade scripts for 3.1
> 
>
> Key: HIVE-19323
> URL: https://issues.apache.org/jira/browse/HIVE-19323
> Project: Hive
>  Issue Type: Task
>  Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-19323.2.patch, HIVE-19323.patch
>
>
> Now that we've branched for 3.0 we need to create SQL install and upgrade 
> scripts for 3.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19338) isExplicitAnalyze method may be incorrect in BasicStatsTask

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1642#comment-1642
 ] 

Jesus Camacho Rodriguez commented on HIVE-19338:


+1 pending tests

> isExplicitAnalyze method may be incorrect in BasicStatsTask
> ---
>
> Key: HIVE-19338
> URL: https://issues.apache.org/jira/browse/HIVE-19338
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19338.patch
>
>
> It relies on a specific ctor being used, however this ctor is used on 
> non-analyze paths too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19338) isExplicitAnalyze method may be incorrect in BasicStatsTask

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19338:

Status: Patch Available  (was: Open)

[~jcamachorodriguez] can you take a look? 
Found out while looking at something else.


> isExplicitAnalyze method may be incorrect in BasicStatsTask
> ---
>
> Key: HIVE-19338
> URL: https://issues.apache.org/jira/browse/HIVE-19338
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19338.patch
>
>
> It relies on a specific ctor being used, however this ctor is used on 
> non-analyze paths too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19317) Handle schema evolution from int like types to decimal

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455542#comment-16455542
 ] 

Hive QA commented on HIVE-19317:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
43s{color} | {color:red} ql: The patch generated 1 new + 19 unchanged - 0 fixed 
= 20 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m  5s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10514/dev-support/hive-personality.sh
 |
| git revision | master / fc42593 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10514/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10514/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Handle schema evolution from int like types to decimal
> --
>
> Key: HIVE-19317
> URL: https://issues.apache.org/jira/browse/HIVE-19317
> Project: Hive
>  Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-19317.1.patch
>
>
> If int like type is changed to decimal on parquet data, select results in 
> errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19338) isExplicitAnalyze method may be incorrect in BasicStatsTask

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19338:

Attachment: HIVE-19338.patch

> isExplicitAnalyze method may be incorrect in BasicStatsTask
> ---
>
> Key: HIVE-19338
> URL: https://issues.apache.org/jira/browse/HIVE-19338
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19338.patch
>
>
> It relies on a specific ctor being used, however this ctor is used on 
> non-analyze paths too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19338) isExplicitAnalyze method may be incorrect in BasicStatsTask

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19338:
---


> isExplicitAnalyze method may be incorrect in BasicStatsTask
> ---
>
> Key: HIVE-19338
> URL: https://issues.apache.org/jira/browse/HIVE-19338
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> It relies on a specific ctor being used, however this ctor is used on 
> non-analyze paths too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19326:

Priority: Blocker  (was: Major)

> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Found when investigating the results change after converting tables to MM, 
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 5   
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 15
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles2   
>   numRows 20  
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate, 
> but  BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect 
> results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19326:

Fix Version/s: 3.0.0

> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
>Priority: Blocker
> Fix For: 3.0.0
>
>
> Found when investigating the results change after converting tables to MM, 
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 5   
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 15
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles2   
>   numRows 20  
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate, 
> but  BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect 
> results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

[Aaron Fabbri (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455520#comment-16455520
 ] 

Aaron Fabbri edited comment on HIVE-16295 at 4/26/18 11:08 PM:
---

This is a really cool prototype [~stakiar], thank you for doing this. I don't 
have much Hive knowledge but will try to spend some more time looking at the 
code.  I'm also happy to work w/ [~ste...@apache.org] on stabilizing the 
_SUCCESS file manifest (which enumerates the files committed) if that works for 
your dynamic partitioning problem.

edit: need more coffee.


was (Author: fabbri):
This is a really cool prototype [~stakiar], thank you for doing this. I don't 
have much Hive knowledge but will try to spend some more time looking at the 
code.  I'm also happy to work w/ [~ste...@apache.org] on stabilizing the 
_SUCCESS file manifest (which enumerates the uploaded-but-not-completed 
multipart uploads to S3) if that works for your dynamic partitioning problem.

> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a 
> {{NullOutputCommitter}} and uses its own commit logic spread across 
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with 
> S3Guard and does a safe, coordinate commit of data on S3 inside individual 
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}} 
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means 
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from 
> task retries or speculative execution) should not step on each other



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter

[Aaron Fabbri (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455520#comment-16455520
 ] 

Aaron Fabbri commented on HIVE-16295:
-

This is a really cool prototype [~stakiar], thank you for doing this. I don't 
have much Hive knowledge but will try to spend some more time looking at the 
code.  I'm also happy to work w/ [~ste...@apache.org] on stabilizing the 
_SUCCESS file manifest (which enumerates the uploaded-but-not-completed 
multipart uploads to S3) if that works for your dynamic partitioning problem.

> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a 
> {{NullOutputCommitter}} and uses its own commit logic spread across 
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with 
> S3Guard and does a safe, coordinate commit of data on S3 inside individual 
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}} 
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means 
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from 
> task retries or speculative execution) should not step on each other



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)

[Eric Wohlstadter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-19305 started by Eric Wohlstadter.
---
> Arrow format for LlapOutputFormatService (umbrella)
> ---
>
> Key: HIVE-19305
> URL: https://issues.apache.org/jira/browse/HIVE-19305
> Project: Hive
>  Issue Type: Improvement
>  Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> Allows external clients to consume output from LLAP daemons in Arrow stream 
> format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19211) New streaming ingest API and support for dynamic partitioning

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455510#comment-16455510
 ] 

Prasanth Jayachandran commented on HIVE-19211:
--

- Simplified partition handling
- Simplified heartbeat
- Addressed review comments

> New streaming ingest API and support for dynamic partitioning
> -
>
> Key: HIVE-19211
> URL: https://issues.apache.org/jira/browse/HIVE-19211
> Project: Hive
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19211.1.patch, HIVE-19211.2.patch, 
> HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch, 
> HIVE-19211.6.patch, HIVE-19211.7.patch, HIVE-19211.8.patch
>
>
> - New streaming API under new hive sub-module
> - Dynamic partitioning support
> - Auto-rollover transactions
> - Automatic heartbeating



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18881) Lower Logging for FSStatsAggregator

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455508#comment-16455508
 ] 

Hive QA commented on HIVE-18881:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920813/HIVE-18881.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 14280 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb] 
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
 (batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
 (batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
 (batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
 (batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
 (batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
 (batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg 
(batchId=254)

[jira] [Updated] (HIVE-19211) New streaming ingest API and support for dynamic partitioning

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-19211:
-
Attachment: HIVE-19211.8.patch

> New streaming ingest API and support for dynamic partitioning
> -
>
> Key: HIVE-19211
> URL: https://issues.apache.org/jira/browse/HIVE-19211
> Project: Hive
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19211.1.patch, HIVE-19211.2.patch, 
> HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch, 
> HIVE-19211.6.patch, HIVE-19211.7.patch, HIVE-19211.8.patch
>
>
> - New streaming API under new hive sub-module
> - Dynamic partitioning support
> - Auto-rollover transactions
> - Automatic heartbeating



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients

[Eric Wohlstadter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-19308:

Attachment: (was: HIVE-19308.1.patch)

> Provide an Arrow stream reader for external LLAP clients 
> -
>
> Key: HIVE-19308
> URL: https://issues.apache.org/jira/browse/HIVE-19308
> Project: Hive
>  Issue Type: Task
>  Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19308.1.patch
>
>
> This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream 
> and produces Arrow batches for an external client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients

[Eric Wohlstadter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-19308:

Attachment: HIVE-19308.1.patch

> Provide an Arrow stream reader for external LLAP clients 
> -
>
> Key: HIVE-19308
> URL: https://issues.apache.org/jira/browse/HIVE-19308
> Project: Hive
>  Issue Type: Task
>  Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19308.1.patch
>
>
> This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream 
> and produces Arrow batches for an external client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients

[Eric Wohlstadter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-19308:

Attachment: HIVE-19308.1.patch

> Provide an Arrow stream reader for external LLAP clients 
> -
>
> Key: HIVE-19308
> URL: https://issues.apache.org/jira/browse/HIVE-19308
> Project: Hive
>  Issue Type: Task
>  Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19308.1.patch
>
>
> This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream 
> and produces Arrow batches for an external client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients

[Eric Wohlstadter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-19308 started by Eric Wohlstadter.
---
> Provide an Arrow stream reader for external LLAP clients 
> -
>
> Key: HIVE-19308
> URL: https://issues.apache.org/jira/browse/HIVE-19308
> Project: Hive
>  Issue Type: Task
>  Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19308.1.patch
>
>
> This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream 
> and produces Arrow batches for an external client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19337) Partition whitelist regex doesn't work (and never did)

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-19337:
--
Status: Patch Available  (was: Open)

> Partition whitelist regex doesn't work (and never did)
> --
>
> Key: HIVE-19337
> URL: https://issues.apache.org/jira/browse/HIVE-19337
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-19337.01.branch-2.patch
>
>
> {{ObjectStore.setConf()}} has the following code:
> {code:java}
> String partitionValidationRegex =
>  
> hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
> {code}
>  Note that it uses name() method which returns enum name 
> (METASTORE_PARTITION_NAME_WHITELIST_PATTERN) rather then .varname
> As a result the regex will always be null.
> The code was introduced as part of 
> HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions
> So looks like this was broken since the original code drop. This is fixed in 
> Hive3 - probably when [~alangates] reworked access to configuration 
> (HIVE-17733) so it isn't a bug in Hive-3.
> [~stakiar_impala_496e] FYI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19337) Partition whitelist regex doesn't work (and never did)

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-19337:
--
Attachment: HIVE-19337.01.branch-2.patch

> Partition whitelist regex doesn't work (and never did)
> --
>
> Key: HIVE-19337
> URL: https://issues.apache.org/jira/browse/HIVE-19337
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-19337.01.branch-2.patch
>
>
> {{ObjectStore.setConf()}} has the following code:
> {code:java}
> String partitionValidationRegex =
>  
> hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
> {code}
>  Note that it uses name() method which returns enum name 
> (METASTORE_PARTITION_NAME_WHITELIST_PATTERN) rather then .varname
> As a result the regex will always be null.
> The code was introduced as part of 
> HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions
> So looks like this was broken since the original code drop. This is fixed in 
> Hive3 - probably when [~alangates] reworked access to configuration 
> (HIVE-17733) so it isn't a bug in Hive-3.
> [~stakiar_impala_496e] FYI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19337) Partition whitelist regex doesn't work (and never did)

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-19337:
--
Description: 
{{ObjectStore.setConf()}} has the following code:
{code:java}
String partitionValidationRegex =
 
hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
{code}
 Note that it uses name() method which returns enum name 
(METASTORE_PARTITION_NAME_WHITELIST_PATTERN) rather then .varname

As a result the regex will always be null.

The code was introduced as part of 

HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions

So looks like this was broken since the original code drop. This is fixed in 
Hive3 - probably when [~alangates] reworked access to configuration 
(HIVE-17733) so it isn't a bug in Hive-3.

[~stakiar_impala_496e] FYI.

  was:
{{ObjectStore.setConf()}} has the following code:
{code:java}
String partitionValidationRegex =
 
hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
{code}
 Note that it uses {{name()}}method which returns enum name 
({{METASTORE_PARTITION_NAME_WHITELIST_PATTERN}}) rather then {.varname}

As a result the regex will always be null.

The code was introduced as part of 

HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions

So looks like this was broken since the original code drop. This is fixed in 
Hive3 - probably when [~alangates] reworked access to configuration 
(HIVE-17733) so it isn't a bug in Hive-3.

[~stakiar_impala_496e] FYI.


> Partition whitelist regex doesn't work (and never did)
> --
>
> Key: HIVE-19337
> URL: https://issues.apache.org/jira/browse/HIVE-19337
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
>
> {{ObjectStore.setConf()}} has the following code:
> {code:java}
> String partitionValidationRegex =
>  
> hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
> {code}
>  Note that it uses name() method which returns enum name 
> (METASTORE_PARTITION_NAME_WHITELIST_PATTERN) rather then .varname
> As a result the regex will always be null.
> The code was introduced as part of 
> HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions
> So looks like this was broken since the original code drop. This is fixed in 
> Hive3 - probably when [~alangates] reworked access to configuration 
> (HIVE-17733) so it isn't a bug in Hive-3.
> [~stakiar_impala_496e] FYI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19337) Partition whitelist regex doesn't work (and never did)

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov reassigned HIVE-19337:
-


> Partition whitelist regex doesn't work (and never did)
> --
>
> Key: HIVE-19337
> URL: https://issues.apache.org/jira/browse/HIVE-19337
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
>
> {{ObjectStore.setConf()}} has the following code:
> {code:java}
> String partitionValidationRegex =
>  
> hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
> {code}
>  Note that it uses {{name()}}method which returns enum name 
> ({{METASTORE_PARTITION_NAME_WHITELIST_PATTERN}}) rather then {.varname}
> As a result the regex will always be null.
> The code was introduced as part of 
> HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions
> So looks like this was broken since the original code drop. This is fixed in 
> Hive3 - probably when [~alangates] reworked access to configuration 
> (HIVE-17733) so it isn't a bug in Hive-3.
> [~stakiar_impala_496e] FYI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19330) multi_insert_partitioned.q fails with "src table does not exist" message.

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455450#comment-16455450
 ] 

Jason Dere commented on HIVE-19330:
---

+1. Apparently this does not require a golden file change

> multi_insert_partitioned.q fails with "src table does not exist" message.
> -
>
> Key: HIVE-19330
> URL: https://issues.apache.org/jira/browse/HIVE-19330
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-19330.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19330) multi_insert_partitioned.q fails with "src table does not exist" message.

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-19330:
--
Status: Patch Available  (was: Open)

> multi_insert_partitioned.q fails with "src table does not exist" message.
> -
>
> Key: HIVE-19330
> URL: https://issues.apache.org/jira/browse/HIVE-19330
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-19330.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19231) Beeline generates garbled output when using UnsupportedTerminal

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455449#comment-16455449
 ] 

Aihua Xu commented on HIVE-19231:
-

? for TTY means the process is not attached to any TTY (terminal interface). 
Probably similar to something like daemon. Seems we should add such check for 
background process. I also notice some platform on mac may show as "??". The 
script works for both case, right?

Probably you can try out cron job calling beeline to simulate this. 

 

> Beeline generates garbled output when using UnsupportedTerminal
> ---
>
> Key: HIVE-19231
> URL: https://issues.apache.org/jira/browse/HIVE-19231
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 2.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-19231.patch
>
>
> We had a customer that was using some sort of front end that would invoke 
> beeline commands with some query files on a node that that remote to the HS2 
> node.
> So beeline runs locally on this edge but connects to a remote HS2. Since the 
> fix made in HIVE-14342, the beeline started producing garbled line in the 
> output. Something like
> {code:java}
> ^Mnull   ^Mnull^Mnull 
>   ^Mnull00- All Occupations 
> 135185230   42270
> 11-   Management occupations  6152650 100310{code}
>  
> I havent been able to reproduce the issue locally as I do not have their 
> system, but with some additional instrumentation I have been able to get some 
> info regarding the beeline process.
> Essentially, such invocation causes beeline process to run with 
> {{-Djline.terminal=jline.UnsupportedTerminal}} all the time and thus causes 
> the issue. They can run the same beeline command directly in the shell on the 
> same host and it does not cause this issue.
> PID    S   TTY  TIME COMMAND
> 44107  S    S  ?    00:00:00 bash beeline -u ...
> PID  S TTY  TIME COMMAND
> 48453  S+   S pts/4    00:00:00 bash beeline -u ...
> Somehow that process wasnt attached to any local terminals. So the check made 
> for /dev/stdin wouldnt work.
>  
> Instead an additional check to check the TTY session of the process before 
> using the UnsupportedTerminal (which really should only be used for 
> backgrounded beeline sessions) seems to resolve the issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18881) Lower Logging for FSStatsAggregator

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455363#comment-16455363
 ] 

Hive QA commented on HIVE-18881:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10513/dev-support/hive-personality.sh
 |
| git revision | master / fc42593 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10513/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Lower Logging for FSStatsAggregator
> ---
>
> Key: HIVE-18881
> URL: https://issues.apache.org/jira/browse/HIVE-18881
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
>  Labels: noob
> Attachments: HIVE-18881.2.patch, HIVE-18881.patch
>
>
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
> {code:java}
>   LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
> {code}
>  # All the other logging in this class is _debug_ or _error_ level logging. 
> This should be _debug_ as well
>  # Remove tab characters to allow splitting on tabs in any kind of 
> tab-separated file of log lines
>  # Use SLF4J parameterized logging



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455227#comment-16455227
 ] 

Eugene Koifman commented on HIVE-19322:
---

the same change doesn't fix these tests so it's not caused by  HIVE-18739.

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18958) Fix Spark config warnings

[Bharathkrishna Guruvayoor Murali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharathkrishna Guruvayoor Murali updated HIVE-18958:

Attachment: HIVE-18958.03.patch

> Fix Spark config warnings
> -
>
> Key: HIVE-18958
> URL: https://issues.apache.org/jira/browse/HIVE-18958
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18958.01.patch, HIVE-18958.02.patch, 
> HIVE-18958.03.patch
>
>
> Getting a few configuration warnings in the logs that we should fix:
> {code}
> 2018-03-14T10:06:19,164  WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
> spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has 
> been deprecated as of Spark 2.3 and may be removed in the future. Please use 
> the new key 'spark.driver.memoryOverhead' instead.
> 2018-03-14T10:06:19,165  WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
> spark.SparkConf: The configuration key spark.akka.logLifecycleEvents is not 
> supported any more because Spark doesn't use Akka since 2.0
> 2018-03-14T10:06:19,165  WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
> spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' 
> has been deprecated as of Spark 2.3 and may be removed in the future. Please 
> use the new key 'spark.executor.memoryOverhead' instead.
> 2018-03-14T10:06:20,351  INFO 
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
> client.SparkClientImpl: Warning: Ignoring non-spark config property: 
> hive.spark.client.server.connect.timeout=9
> 2018-03-14T10:06:20,351  INFO 
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
> client.SparkClientImpl: Warning: Ignoring non-spark config property: 
> hive.spark.client.rpc.threads=8
> 2018-03-14T10:06:20,351  INFO 
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
> client.SparkClientImpl: Warning: Ignoring non-spark config property: 
> hive.spark.client.connect.timeout=3
> 2018-03-14T10:06:20,351  INFO 
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
> client.SparkClientImpl: Warning: Ignoring non-spark config property: 
> hive.spark.client.secret.bits=256
> 2018-03-14T10:06:20,351  INFO 
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main] 
> client.SparkClientImpl: Warning: Ignoring non-spark config property: 
> hive.spark.client.rpc.max.size=52428800
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19228) Remove commons-httpclient 3.x usage

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454900#comment-16454900
 ] 

Hive QA commented on HIVE-19228:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920817/HIVE-19228.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 14280 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb] 
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
 (batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
 (batchId=98)
org.apache.hadoop.hive.metastore.client.TestNotNullConstraint.createTableWithConstraintsPkInOtherCatalog[Remote]
 (batchId=211)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
 (batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
 (batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill 
(batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection 
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg 
(batchId=254)

[jira] [Updated] (HIVE-19332) Disable compute.query.using.stats for external table

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-19332:
--
Description: Hive can use statistics to answer queries like count(*). This 
can be problematic on external tables where another tool might add files that 
Hive doesn’t know about. In that case Hive will return incorrect results.  
(was: Hive can use statistics to answer queries like “count(*)”. This can be 
problematic on external tables where another tool might add files that Hive 
doesn’t know about. In that case Hive will return incorrect results.)

> Disable compute.query.using.stats for external table
> 
>
> Key: HIVE-19332
> URL: https://issues.apache.org/jira/browse/HIVE-19332
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jason Dere
>Priority: Major
>
> Hive can use statistics to answer queries like count(*). This can be 
> problematic on external tables where another tool might add files that Hive 
> doesn’t know about. In that case Hive will return incorrect results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19332) Disable compute.query.using.stats for external table

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-19332:
--
Description: Hive can use statistics to answer queries like “count(*)”. 
This can be problematic on external tables where another tool might add files 
that Hive doesn’t know about. In that case Hive will return incorrect results.

> Disable compute.query.using.stats for external table
> 
>
> Key: HIVE-19332
> URL: https://issues.apache.org/jira/browse/HIVE-19332
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jason Dere
>Priority: Major
>
> Hive can use statistics to answer queries like “count(*)”. This can be 
> problematic on external tables where another tool might add files that Hive 
> doesn’t know about. In that case Hive will return incorrect results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19330) multi_insert_partitioned.q fails with "src table does not exist" message.

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454893#comment-16454893
 ] 

Steve Yeom commented on HIVE-19330:
---

Hi [~jdere] can you help me upload this one line patch?
Thanks, 
Steve. 

> multi_insert_partitioned.q fails with "src table does not exist" message.
> -
>
> Key: HIVE-19330
> URL: https://issues.apache.org/jira/browse/HIVE-19330
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-19330.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19330) multi_insert_partitioned.q fails with "src table does not exist" message.

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454890#comment-16454890
 ] 

Vineet Garg commented on HIVE-19330:


cool. Your patch needs to update golden file too.

> multi_insert_partitioned.q fails with "src table does not exist" message.
> -
>
> Key: HIVE-19330
> URL: https://issues.apache.org/jira/browse/HIVE-19330
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-19330.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19330) multi_insert_partitioned.q fails with "src table does not exist" message.

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-19330:
--
Attachment: HIVE-19330.01.patch

> multi_insert_partitioned.q fails with "src table does not exist" message.
> -
>
> Key: HIVE-19330
> URL: https://issues.apache.org/jira/browse/HIVE-19330
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-19330.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19330) multi_insert_partitioned.q fails with "src table does not exist" message.

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom reassigned HIVE-19330:
-

Assignee: Steve Yeom

> multi_insert_partitioned.q fails with "src table does not exist" message.
> -
>
> Key: HIVE-19330
> URL: https://issues.apache.org/jira/browse/HIVE-19330
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19330) multi_insert_partitioned.q fails with "src table does not exist" message.

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454882#comment-16454882
 ] 

Steve Yeom commented on HIVE-19330:
---

yes i have a patch so i created this jira.

> multi_insert_partitioned.q fails with "src table does not exist" message.
> -
>
> Key: HIVE-19330
> URL: https://issues.apache.org/jira/browse/HIVE-19330
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19330) multi_insert_partitioned.q fails with "src table does not exist" message.

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454881#comment-16454881
 ] 

Vineet Garg commented on HIVE-19330:


[~steveyeom2017] you just need to include {{--! qt:dataset:src}} in your q file 
and it should be good.

> multi_insert_partitioned.q fails with "src table does not exist" message.
> -
>
> Key: HIVE-19330
> URL: https://issues.apache.org/jira/browse/HIVE-19330
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19228) Remove commons-httpclient 3.x usage

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454873#comment-16454873
 ] 

Hive QA commented on HIVE-19228:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
55s{color} | {color:green} root: The patch generated 0 new + 68 unchanged - 4 
fixed = 68 total (was 72) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
17s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 19 
unchanged - 1 fixed = 19 total (was 20) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} service: The patch generated 0 new + 39 unchanged - 
3 fixed = 39 total (was 42) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 26s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10512/dev-support/hive-personality.sh
 |
| git revision | master / fc42593 |
| Default Java | 1.8.0_111 |
| modules | C: . itests/hive-unit ql service U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10512/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Remove commons-httpclient 3.x usage
> ---
>
> Key: HIVE-19228
> URL: https://issues.apache.org/jira/browse/HIVE-19228
> Project: Hive
>  Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-19228.1.patch, HIVE-19228.2.patch, 
> HIVE-19228.2.patch, HIVE-19228.2.patch
>
>
> Commons-httpclient is not supported well anymore.  Remove dependency and move 
> to Apache HTTP client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService

[Eric Wohlstadter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-19307 started by Eric Wohlstadter.
---
> Support ArrowOutputStream in LlapOutputFormatService
> 
>
> Key: HIVE-19307
> URL: https://issues.apache.org/jira/browse/HIVE-19307
> Project: Hive
>  Issue Type: Task
>  Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19307.1.patch
>
>
> Support pushing arrow batches through 
> org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService

[Eric Wohlstadter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Wohlstadter updated HIVE-19307:

Attachment: HIVE-19307.1.patch

> Support ArrowOutputStream in LlapOutputFormatService
> 
>
> Key: HIVE-19307
> URL: https://issues.apache.org/jira/browse/HIVE-19307
> Project: Hive
>  Issue Type: Task
>  Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19307.1.patch
>
>
> Support pushing arrow batches through 
> org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19323) Create metastore SQL install and upgrade scripts for 3.1

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-19323:
--
Status: Patch Available  (was: Open)

> Create metastore SQL install and upgrade scripts for 3.1
> 
>
> Key: HIVE-19323
> URL: https://issues.apache.org/jira/browse/HIVE-19323
> Project: Hive
>  Issue Type: Task
>  Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-19323.patch
>
>
> Now that we've branched for 3.0 we need to create SQL install and upgrade 
> scripts for 3.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19323) Create metastore SQL install and upgrade scripts for 3.1

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-19323:
--
Attachment: HIVE-19323.patch

> Create metastore SQL install and upgrade scripts for 3.1
> 
>
> Key: HIVE-19323
> URL: https://issues.apache.org/jira/browse/HIVE-19323
> Project: Hive
>  Issue Type: Task
>  Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HIVE-19323.patch
>
>
> Now that we've branched for 3.0 we need to create SQL install and upgrade 
> scripts for 3.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19318) Improve Hive logging

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-19318:

Description: Use this jira to track some potential improvements on hive 
logging. What I have noticed that some log entries may have incorrect log 
level, or may not show in the correct places, e.g., some logs are printing to 
the STDERR/STDOUT rather than the HS2 log file.   (was: Use this jira to track 
some potential improvements on hive logging. What I have noticed that some log 
entries may have incorrect log level, or may not show in the correct places, 
e.g., printing to the STDERR/STDOUT rather than the HS2 log file. )

> Improve Hive logging
> 
>
> Key: HIVE-19318
> URL: https://issues.apache.org/jira/browse/HIVE-19318
> Project: Hive
>  Issue Type: Improvement
>  Components: Logging
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>
> Use this jira to track some potential improvements on hive logging. What I 
> have noticed that some log entries may have incorrect log level, or may not 
> show in the correct places, e.g., some logs are printing to the STDERR/STDOUT 
> rather than the HS2 log file. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19331) Repl load config in "with" clause not pass to Context.getStagingDir

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-19331:
--
Attachment: HIVE-19331.1.patch

> Repl load config in "with" clause not pass to Context.getStagingDir
> ---
>
> Key: HIVE-19331
> URL: https://issues.apache.org/jira/browse/HIVE-19331
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19331.1.patch
>
>
> Another failure similar to HIVE-18626, causing exception when s3 credentials 
> are in "REPL LOAD" with clause.
> {code}
> Caused by: java.lang.IllegalStateException: Error getting FileSystem for 
> s3a://nat-yc-r7-nmys-beacon-cloud-s3-2/hive_incremental_testing.db/hive_incremental_testing_new_tabl...:
>  org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on 
> nat-yc-r7-nmys-beacon-cloud-s3-2: com.amazonaws.AmazonClientException: No AWS 
> Credentials provided by BasicAWSCredentialsProvider 
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
> EC2 metadata service: No AWS Credentials provided by 
> BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
> EC2 metadata service
> at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:359)
> at 
> org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:487)
> at 
> org.apache.hadoop.hive.ql.Context.getExternalTmpPath(Context.java:565)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.loadTable(ImportSemanticAnalyzer.java:370)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.createReplImportTasks(ImportSemanticAnalyzer.java:926)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(ImportSemanticAnalyzer.java:329)
> at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:43)
> ... 24 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19331) Repl load config in "with" clause not pass to Context.getStagingDir

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-19331:
--
Status: Patch Available  (was: Open)

> Repl load config in "with" clause not pass to Context.getStagingDir
> ---
>
> Key: HIVE-19331
> URL: https://issues.apache.org/jira/browse/HIVE-19331
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19331.1.patch
>
>
> Another failure similar to HIVE-18626, causing exception when s3 credentials 
> are in "REPL LOAD" with clause.
> {code}
> Caused by: java.lang.IllegalStateException: Error getting FileSystem for 
> s3a://nat-yc-r7-nmys-beacon-cloud-s3-2/hive_incremental_testing.db/hive_incremental_testing_new_tabl...:
>  org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on 
> nat-yc-r7-nmys-beacon-cloud-s3-2: com.amazonaws.AmazonClientException: No AWS 
> Credentials provided by BasicAWSCredentialsProvider 
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
> EC2 metadata service: No AWS Credentials provided by 
> BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
> EC2 metadata service
> at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:359)
> at 
> org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:487)
> at 
> org.apache.hadoop.hive.ql.Context.getExternalTmpPath(Context.java:565)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.loadTable(ImportSemanticAnalyzer.java:370)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.createReplImportTasks(ImportSemanticAnalyzer.java:926)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(ImportSemanticAnalyzer.java:329)
> at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:43)
> ... 24 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19331) Repl load config in "with" clause not pass to Context.getStagingDir

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-19331:
-


> Repl load config in "with" clause not pass to Context.getStagingDir
> ---
>
> Key: HIVE-19331
> URL: https://issues.apache.org/jira/browse/HIVE-19331
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
>
> Another failure similar to HIVE-18626, causing exception when s3 credentials 
> are in "REPL LOAD" with clause.
> {code}
> Caused by: java.lang.IllegalStateException: Error getting FileSystem for 
> s3a://nat-yc-r7-nmys-beacon-cloud-s3-2/hive_incremental_testing.db/hive_incremental_testing_new_tabl...:
>  org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on 
> nat-yc-r7-nmys-beacon-cloud-s3-2: com.amazonaws.AmazonClientException: No AWS 
> Credentials provided by BasicAWSCredentialsProvider 
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
> EC2 metadata service: No AWS Credentials provided by 
> BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon 
> EC2 metadata service
> at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:359)
> at 
> org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:487)
> at 
> org.apache.hadoop.hive.ql.Context.getExternalTmpPath(Context.java:565)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.loadTable(ImportSemanticAnalyzer.java:370)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.createReplImportTasks(ImportSemanticAnalyzer.java:926)
> at 
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(ImportSemanticAnalyzer.java:329)
> at 
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:43)
> ... 24 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19329) Disallow some optimizations/behaviors for external tables

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere reassigned HIVE-19329:
-


> Disallow some optimizations/behaviors for external tables
> -
>
> Key: HIVE-19329
> URL: https://issues.apache.org/jira/browse/HIVE-19329
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> External tables in Hive are often used in situations where the data is being 
> created and managed by other applications outside of Hive. There are several 
> issues that can occur when data being written to table directories by 
> external apps:
> - If an application is writing files to a table/partition at the same time 
> that Hive tries to merge files for the same table/partition (ALTER TABLE 
> CONCATENATE, or hive.merge.tezfiles during insert) data can be lost.
> - When new data has been added to the table by external applications, the 
> Hive table statistics are often way out of date with the current state of the 
> data. This can result in wrong results in the case of answering queries using 
> stats, or bad query plans being generated.
> Some of these operations should be blocked in Hive. It looks like some 
> already have been (HIVE-17403).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454861#comment-16454861
 ] 

Vineet Garg commented on HIVE-19322:


[~ekoifman] minimrclidriver has the following extra information in output
{noformat}
< Error during job, obtaining debugging information...
< Job Tracking URL: 
http://hive-ptest-slaves-7f.c.gcp-hive-upstream.internal:33378/proxy/application_1524737839942_0003/
< Examining task ID: task_#ID#m_00 (and more) from job 
job_1524737839942_0003
< 
< Task with the most failures(4): 
< -
< Task ID:
<   task_#ID#m_00
< 
< URL:
<   
http://hive-ptest-slaves-7f.c.gcp-hive-upstream.internal:33378/taskdetails.jsp?jobid=job_1524737839942_0003=task_#ID#m_00
< -
{noformat}

The tests i have listed have the following extra information in output:
{noformat}
< Error during job, obtaining debugging information...
{noformat}

The only difference I see is TestNegativeMinimrCliDriver prints actual 
debugging information and rest of the tests doesn't. I assumed HIVE-18739 has 
made changes to cause this. Correct me if I am wrong.

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454854#comment-16454854
 ] 

Eugene Koifman commented on HIVE-19322:
---

[~vgarg], I checked the tests you posted - not 1 of them has the same error...

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
Affects Version/s: 3.0.0
   Status: Patch Available  (was: Open)

this is a test only change.
Something changed in compiler/runtime that writes rows in different orders so 
they get row_ids assigned in a different order.  Change is not significant.

[~sankarh] could you review please

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19315.01.patch
>
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
Attachment: HIVE-19315.01.patch

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19315.01.patch
>
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19327) qroupby_rollup_empty.q fails for insert-only transactional tables

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-19327:
--
Status: Patch Available  (was: Open)

> qroupby_rollup_empty.q fails for insert-only transactional tables
> -
>
> Key: HIVE-19327
> URL: https://issues.apache.org/jira/browse/HIVE-19327
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19327.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19327) qroupby_rollup_empty.q fails for insert-only transactional tables

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-19327:
--
Attachment: HIVE-19327.01.patch

> qroupby_rollup_empty.q fails for insert-only transactional tables
> -
>
> Key: HIVE-19327
> URL: https://issues.apache.org/jira/browse/HIVE-19327
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19327.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19327) qroupby_rollup_empty.q fails for insert-only transactional tables

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom reassigned HIVE-19327:
-

Assignee: Steve Yeom

> qroupby_rollup_empty.q fails for insert-only transactional tables
> -
>
> Key: HIVE-19327
> URL: https://issues.apache.org/jira/browse/HIVE-19327
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19324) improve YARN queue check error message in Tez pool

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454806#comment-16454806
 ] 

Ashutosh Chauhan commented on HIVE-19324:
-

+1

> improve YARN queue check error message in Tez pool
> --
>
> Key: HIVE-19324
> URL: https://issues.apache.org/jira/browse/HIVE-19324
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepesh Khandelwal
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19324.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19326) union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19326:

Summary: union_fast_stats MiniLlapLocal golden file has incorrect 
"accurate" stats  (was: union_fast_stats golden file has incorrect "accurate" 
stats)

> union_fast_stats MiniLlapLocal golden file has incorrect "accurate" stats
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> Found when investigating the results change after converting tables to MM, 
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 5   
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 15
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles2   
>   numRows 20  
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate, 
> but  BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect 
> results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19326) union_fast_stats golden file has incorrect "accurate" stats

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19326:

Description: 
Found when investigating the results change after converting tables to MM, 
turns out the MM result is correct but the current one is not.
The test ends like so:
{noformat}
desc formatted small_alltypesorc_a;
ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
desc formatted small_alltypesorc_a;
insert into table small_alltypesorc_a select * from small_alltypesorc1a;
desc formatted small_alltypesorc_a;
{noformat}

The results from the descs in the golden file are:
{noformat}
COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
numFiles1   
numRows 5   
...
COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
numFiles1   
numRows 15
...
COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
numFiles2   
numRows 20  
{noformat}

Note the result change after analyze - the original nomRows is inaccurate, but  
BASIC_STATS is set to true.

I am assuming with metadata only optimization this can produce incorrect 
results.

  was:
Found when investigating results change after converting tables to MM, turns 
out the MM result is correct but the current one is not.
The test ends like so:
{noformat}
desc formatted small_alltypesorc_a;
ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
desc formatted small_alltypesorc_a;
insert into table small_alltypesorc_a select * from small_alltypesorc1a;
desc formatted small_alltypesorc_a;
{noformat}

The results from the descs in the golden file are:
{noformat}
COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
numFiles1   
numRows 5   
...
COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
numFiles1   
numRows 15
...
COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
numFiles2   
numRows 20  
{noformat}

Note the result change after analyze - the original nomRows is inaccurate, but  
BASIC_STATS is set to true.

I am assuming with metadata only optimization this can produce incorrect 
results.


> union_fast_stats golden file has incorrect "accurate" stats
> ---
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> Found when investigating the results change after converting tables to MM, 
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 5   
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 15
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles2   
>   numRows 20  
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate, 
> but  BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect 
> results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454804#comment-16454804
 ] 

Eugene Koifman commented on HIVE-19124:
---

+1

> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, 
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, 
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, 
> HIVE-19124.08.patch, HIVE-19124.09.patch, HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be 
> supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19326) union_fast_stats golden file has incorrect "accurate" stats

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454801#comment-16454801
 ] 

Sergey Shelukhin commented on HIVE-19326:
-

[~ashutoshc] [~prasanth_j] looks like a stats issue that may cause problems 
with metadata only queries.
Can you confirm the latter part (ie whether this is important if 
BASIC_STATS=true but numRows is wrong).

> union_fast_stats golden file has incorrect "accurate" stats
> ---
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> Found when investigating results change after converting tables to MM, turns 
> out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 5   
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 15
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles2   
>   numRows 20  
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate, 
> but  BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect 
> results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19326) union_fast_stats golden file has incorrect "accurate" stats

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19326:
---


> union_fast_stats golden file has incorrect "accurate" stats
> ---
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> Found when investigating results change after converting tables to MM, turns 
> out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 5   
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles1   
>   numRows 15
> ...
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
>   numFiles2   
>   numRows 20  
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate, 
> but  BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect 
> results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19325) Custom Hive Patch - Remove beeline -n flag in Hive 0.13.1

[Alejandro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Fernandez updated HIVE-19325:
---
Description: 
This Jira is not meant to be contributed back, but I'm using it as a way to run 
unit tests against a patch file.
Specifically, TestBeelineWithArgs, ProxyAuthTest, and TestSchemaTool

Remove beeline -n flag used for impersonation.

  was:
This Jira is not meant to be contributed back, but I'm using it as a way to run 
unit tests against a patch file.

Remove beeline -n flag used for impersonation.


> Custom Hive Patch - Remove beeline -n flag in Hive 0.13.1
> -
>
> Key: HIVE-19325
> URL: https://issues.apache.org/jira/browse/HIVE-19325
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 0.13.1
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
>Priority: Major
> Fix For: 0.13.1
>
> Attachments: HIVE-19325-0.13.1.patch, HIVE-19325-branch-0.13.1.patch, 
> HIVE-19325.0.13.1.patch, HIVE-19325.branch-0.13.1.patch
>
>
> This Jira is not meant to be contributed back, but I'm using it as a way to 
> run unit tests against a patch file.
> Specifically, TestBeelineWithArgs, ProxyAuthTest, and TestSchemaTool
> Remove beeline -n flag used for impersonation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18903) Lower Logging Level for ObjectStore

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454776#comment-16454776
 ] 

Hive QA commented on HIVE-18903:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920812/HIVE-18903.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 14280 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
 (batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
 (batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
 (batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
 (batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
 (batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel 
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
 (batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
 (batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
 (batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles 
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent 
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead 
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes 
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
 (batchId=242)