[jira] [Commented] (HIVE-17760) Create a unit test which validates HIVE-9423 does not regress

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201485#comment-16201485
 ] 

Hive QA commented on HIVE-17760:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891550/HIVE-17760.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 11212 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
org.apache.hive.beeline.TestBeelinePasswordOption.testMultiConnect (batchId=222)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7245/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7245/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7245/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12891550 - PreCommit-HIVE-Build

> Create a unit test which validates HIVE-9423 does not regress 
> --
>
> Key: HIVE-17760
> URL: https://issues.apache.org/jira/browse/HIVE-17760
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-17760.1.patch, HIVE-17760.2.patch
>
>
> During [HIVE-9423] we verified that when the Thrift server pool is exhausted, 
> then Beeline connection times out, and provide a meaningful error message.
> Create a unit test which verifies this, and helps to keep this feature working



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17726) Using exists may lead to incorrect results

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-17726:
---
Status: Open  (was: Patch Available)

> Using exists may lead to incorrect results
> --
>
> Key: HIVE-17726
> URL: https://issues.apache.org/jira/browse/HIVE-17726
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Zoltan Haindrich
>Assignee: Vineet Garg
> Attachments: HIVE-17726.1.patch, HIVE-17726.2.patch, 
> HIVE-17726.3.patch
>
>
> {code}
> drop table if exists tx1;
> create table tx1 (a integer,b integer);
> insert into tx1   values  (1, 1),
> (1, 2),
> (1, 3);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b limit 1);
> {code}
> current results are 6 and 2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17726) Using exists may lead to incorrect results

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-17726:
---
Status: Patch Available  (was: Open)

> Using exists may lead to incorrect results
> --
>
> Key: HIVE-17726
> URL: https://issues.apache.org/jira/browse/HIVE-17726
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Zoltan Haindrich
>Assignee: Vineet Garg
> Attachments: HIVE-17726.1.patch, HIVE-17726.2.patch, 
> HIVE-17726.3.patch
>
>
> {code}
> drop table if exists tx1;
> create table tx1 (a integer,b integer);
> insert into tx1   values  (1, 1),
> (1, 2),
> (1, 3);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b limit 1);
> {code}
> current results are 6 and 2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17726) Using exists may lead to incorrect results

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-17726:
---
Attachment: HIVE-17726.3.patch

> Using exists may lead to incorrect results
> --
>
> Key: HIVE-17726
> URL: https://issues.apache.org/jira/browse/HIVE-17726
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Zoltan Haindrich
>Assignee: Vineet Garg
> Attachments: HIVE-17726.1.patch, HIVE-17726.2.patch, 
> HIVE-17726.3.patch
>
>
> {code}
> drop table if exists tx1;
> create table tx1 (a integer,b integer);
> insert into tx1   values  (1, 1),
> (1, 2),
> (1, 3);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b limit 1);
> {code}
> current results are 6 and 2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HIVE-14903) from_utc_time function issue for CET daylight savings

[Eric Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Lin resolved HIVE-14903.
-
Resolution: Fixed

Tested in CDH5.11, confirm fixed, but not sure which JIRA has the fix in the 
upstream, will just resolve it for now.

> from_utc_time function issue for CET daylight savings
> -
>
> Key: HIVE-14903
> URL: https://issues.apache.org/jira/browse/HIVE-14903
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Eric Lin
>Priority: Minor
>
> Based on https://en.wikipedia.org/wiki/Central_European_Summer_Time, the 
> summer time is between 1:00 UTC on the last Sunday of March and 1:00 on the 
> last Sunday of October, see test case below:
> Impala:
> {code}
> select from_utc_timestamp('2016-10-30 00:30:00','CET');
> Query: select from_utc_timestamp('2016-10-30 00:30:00','CET')
> +--+
> | from_utc_timestamp('2016-10-30 00:30:00', 'cet') |
> +--+
> | 2016-10-30 01:30:00  |
> +--+
> {code}
> Hive:
> {code}
> select from_utc_timestamp('2016-10-30 00:30:00','CET');
> INFO  : OK
> ++--+
> |  _c0   |
> ++--+
> | 2016-10-30 01:30:00.0  |
> ++--+
> {code}
> MySQL:
> {code}
> mysql> SELECT CONVERT_TZ( '2016-10-30 00:30:00', 'UTC', 'CET' );
> +---+
> | CONVERT_TZ( '2016-10-30 00:30:00', 'UTC', 'CET' ) |
> +---+
> | 2016-10-30 02:30:00   |
> +---+
> {code}
> At 00:30AM UTC, the daylight saving has not finished so the time different 
> should still be 2 hours rather than 1. MySQL returned correct result
> At 1:30, results are correct:
> Impala:
> {code}
> Query: select from_utc_timestamp('2016-10-30 01:30:00','CET')
> +--+
> | from_utc_timestamp('2016-10-30 01:30:00', 'cet') |
> +--+
> | 2016-10-30 02:30:00  |
> +--+
> Fetched 1 row(s) in 0.01s
> {code}
> Hive:
> {code}
> ++--+
> |  _c0   |
> ++--+
> | 2016-10-30 02:30:00.0  |
> ++--+
> 1 row selected (0.252 seconds)
> {code}
> MySQL:
> {code}
> mysql> SELECT CONVERT_TZ( '2016-10-30 01:30:00', 'UTC', 'CET' );
> +---+
> | CONVERT_TZ( '2016-10-30 01:30:00', 'UTC', 'CET' ) |
> +---+
> | 2016-10-30 02:30:00   |
> +---+
> 1 row in set (0.00 sec)
> {code}
> Seems like a bug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14903) from_utc_time function issue for CET daylight savings

[Eric Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201479#comment-16201479
 ] 

Eric Lin commented on HIVE-14903:
-

Thanks for letting me know [~zsombor.klara], and apologies for the delay.

> from_utc_time function issue for CET daylight savings
> -
>
> Key: HIVE-14903
> URL: https://issues.apache.org/jira/browse/HIVE-14903
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Eric Lin
>Priority: Minor
>
> Based on https://en.wikipedia.org/wiki/Central_European_Summer_Time, the 
> summer time is between 1:00 UTC on the last Sunday of March and 1:00 on the 
> last Sunday of October, see test case below:
> Impala:
> {code}
> select from_utc_timestamp('2016-10-30 00:30:00','CET');
> Query: select from_utc_timestamp('2016-10-30 00:30:00','CET')
> +--+
> | from_utc_timestamp('2016-10-30 00:30:00', 'cet') |
> +--+
> | 2016-10-30 01:30:00  |
> +--+
> {code}
> Hive:
> {code}
> select from_utc_timestamp('2016-10-30 00:30:00','CET');
> INFO  : OK
> ++--+
> |  _c0   |
> ++--+
> | 2016-10-30 01:30:00.0  |
> ++--+
> {code}
> MySQL:
> {code}
> mysql> SELECT CONVERT_TZ( '2016-10-30 00:30:00', 'UTC', 'CET' );
> +---+
> | CONVERT_TZ( '2016-10-30 00:30:00', 'UTC', 'CET' ) |
> +---+
> | 2016-10-30 02:30:00   |
> +---+
> {code}
> At 00:30AM UTC, the daylight saving has not finished so the time different 
> should still be 2 hours rather than 1. MySQL returned correct result
> At 1:30, results are correct:
> Impala:
> {code}
> Query: select from_utc_timestamp('2016-10-30 01:30:00','CET')
> +--+
> | from_utc_timestamp('2016-10-30 01:30:00', 'cet') |
> +--+
> | 2016-10-30 02:30:00  |
> +--+
> Fetched 1 row(s) in 0.01s
> {code}
> Hive:
> {code}
> ++--+
> |  _c0   |
> ++--+
> | 2016-10-30 02:30:00.0  |
> ++--+
> 1 row selected (0.252 seconds)
> {code}
> MySQL:
> {code}
> mysql> SELECT CONVERT_TZ( '2016-10-30 01:30:00', 'UTC', 'CET' );
> +---+
> | CONVERT_TZ( '2016-10-30 01:30:00', 'UTC', 'CET' ) |
> +---+
> | 2016-10-30 02:30:00   |
> +---+
> 1 row in set (0.00 sec)
> {code}
> Seems like a bug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15267) Make query length calculation logic more accurate in TxnUtils.needNewQuery()

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201422#comment-16201422
 ] 

Hive QA commented on HIVE-15267:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891528/HIVE-15267.04.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 11211 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7244/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7244/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7244/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12891528 - PreCommit-HIVE-Build

> Make query length calculation logic more accurate in TxnUtils.needNewQuery()
> 
>
> Key: HIVE-15267
> URL: https://issues.apache.org/jira/browse/HIVE-15267
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Transactions
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Wei Zheng
>Assignee: Steve Yeom
> Attachments: HIVE-15267.01.patch, HIVE-15267.02.patch, 
> HIVE-15267.03.patch, HIVE-15267.04.patch
>
>
> In HIVE-15181 there's such review comment, for which this ticket will handle
> {code}
> in TxnUtils.needNewQuery() "sizeInBytes / 1024 > queryMemoryLimit" doesn't do 
> the right thing.
> If the user sets METASTORE_DIRECT_SQL_MAX_QUERY_LENGTH to 1K, they most 
> likely want each SQL string to be at most 1K.
> But if sizeInBytes=2047, this still returns false.
> It should include length of "suffix" in computation of sizeInBytes
> Along the same lines: the check for max query length is done after each batch 
> is already added to the query. Suppose there are 1000 9-digit txn IDs in each 
> IN(...). That's, conservatively, 18KB of text. So the length of each query is 
> increasing in 18KB chunks. 
> I think the check for query length should be done for each item in IN clause.
> If some DB has a limit on query length of X, then any query > X will fail. So 
> I think this must ensure not to produce any queries > X, even by 1 char.
> For example, case 3.1 of the UT generates a query of almost 4000 characters - 
> this is clearly > 1KB.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201405#comment-16201405
 ] 

Rui Li commented on HIVE-15104:
---

Hi [~xuefuz], sorry for taking so long to update. I tried out your proposal. 
The idea is to build the trivial package and add it with the {{--jars}} config 
when we launch the Spark app. So we need to locate the jar at runtime. To 
locate the jar, we can use reflection to get the class and call 
{{SparkContext.jarOfClass}} to get the URI. But since kryo is shaded, we don't 
have {{com.esotericsoftware.kryo.Kryo}} in our class path on Hive side. When I 
try to get the registrator class, I get a {{NoClassDefFoundError}} for 
{{com.esotericsoftware.kryo.Kryo}}.
I guess one workaround is to let user specify the path to the jar, but that 
seems not very friendly. Any suggestions?

> Hive on Spark generate more shuffle data than hive on mr
> 
>
> Key: HIVE-15104
> URL: https://issues.apache.org/jira/browse/HIVE-15104
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Rui Li
> Attachments: HIVE-15104.1.patch, HIVE-15104.2.patch, 
> HIVE-15104.3.patch, HIVE-15104.4.patch, HIVE-15104.5.patch, 
> HIVE-15104.5.patch, TPC-H 100G.xlsx
>
>
> the same sql,  running on spark  and mr engine, will generate different size 
> of shuffle data.
> i think it is because of hive on mr just serialize part of HiveKey, but hive 
> on spark which using kryo will serialize full of Hivekey object.  
> what is your opionion?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201387#comment-16201387
 ] 

Hive QA commented on HIVE-15016:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891579/HIVE-15016.4.patch

{color:green}SUCCESS:{color} +1 due to 9 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 195 failed/errored test(s), 10499 tests 
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_auto_purge_tables]
 (batchId=166)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_ctas]
 (batchId=166)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_dynamic]
 (batchId=168)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_static]
 (batchId=165)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=169)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
 (batchId=166)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_select_read_only_encrypted_tbl]
 (batchId=167)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_select_read_only_unencrypted_tbl]
 (batchId=167)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
 (batchId=94)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
 (batchId=95)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
 (batchId=96)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
 (batchId=97)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
 (batchId=98)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
 (batchId=99)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[cascade_dbdrop]
 (batchId=237)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[cascade_dbdrop_hadoop20]
 (batchId=237)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[generatehfiles_require_family_path]
 (batchId=237)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver[hbase_ddl] 
(batchId=237)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 

[jira] [Updated] (HIVE-17783) Hybrid Grace Hash Join has performance degradation for N-way join using Hive on Tez

[Ferdinand Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferdinand Xu updated HIVE-17783:

Summary: Hybrid Grace Hash Join has performance degradation for N-way join 
using Hive on Tez  (was: Hybrid Grace Join has performance degradation for 
N-way join using Hive on Tez)

> Hybrid Grace Hash Join has performance degradation for N-way join using Hive 
> on Tez
> ---
>
> Key: HIVE-17783
> URL: https://issues.apache.org/jira/browse/HIVE-17783
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
> Environment: 8*Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
> 1 master + 7 workers
> TPC-DS at 3TB data scales
> Hive version : 2.2.0
>Reporter: Ferdinand Xu
> Attachments: Hybrid_Grace_Hash_Join.xlsx
>
>
> Most configurations are using default value. And the benchmark is to test 
> enabling against disabling hybrid grace hash join using TPC-DS queries at 3TB 
> data scales. Many queries related to N-way join has performance degradation 
> over three times test. Detailed result  is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17783) Hybrid Grace Hash Join has performance degradation for N-way join using Hive on Tez

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201380#comment-16201380
 ] 

Ferdinand Xu commented on HIVE-17783:
-

CC [~wzheng] [~sershe] 

> Hybrid Grace Hash Join has performance degradation for N-way join using Hive 
> on Tez
> ---
>
> Key: HIVE-17783
> URL: https://issues.apache.org/jira/browse/HIVE-17783
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
> Environment: 8*Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
> 1 master + 7 workers
> TPC-DS at 3TB data scales
> Hive version : 2.2.0
>Reporter: Ferdinand Xu
> Attachments: Hybrid_Grace_Hash_Join.xlsx
>
>
> Most configurations are using default value. And the benchmark is to test 
> enabling against disabling hybrid grace hash join using TPC-DS queries at 3TB 
> data scales. Many queries related to N-way join has performance degradation 
> over three times test. Detailed result  is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17783) Hybrid Grace Hash Join has performance degradation for N-way join using Hive on Tez

[Ferdinand Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferdinand Xu updated HIVE-17783:

Attachment: Hybrid_Grace_Hash_Join.xlsx

> Hybrid Grace Hash Join has performance degradation for N-way join using Hive 
> on Tez
> ---
>
> Key: HIVE-17783
> URL: https://issues.apache.org/jira/browse/HIVE-17783
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
> Environment: 8*Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
> 1 master + 7 workers
> TPC-DS at 3TB data scales
> Hive version : 2.2.0
>Reporter: Ferdinand Xu
> Attachments: Hybrid_Grace_Hash_Join.xlsx
>
>
> Most configurations are using default value. And the benchmark is to test 
> enabling against disabling hybrid grace hash join using TPC-DS queries at 3TB 
> data scales. Many queries related to N-way join has performance degradation 
> over three times test. Detailed result  is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17733) Move RawStore to standalone metastore

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201333#comment-16201333
 ] 

Hive QA commented on HIVE-17733:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891497/HIVE-17733.4.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7242/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7242/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7242/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-10-12 02:06:38.348
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-7242/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-10-12 02:06:38.350
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   f1050a6..2c871e4  master -> origin/master
+ git reset --hard HEAD
HEAD is now at f1050a6 HIVE-17669: Cache to optimize SearchArgument 
deserialization (Mithun Radhakrishnan reviewed by Prasanth Jayachandran)
+ git clean -f -d
Removing 
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java.orig
Removing ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g.orig
Removing ql/src/java/org/apache/hadoop/hive/ql/plan/CreateResourcePlanDesc.java
Removing ql/src/java/org/apache/hadoop/hive/ql/plan/ShowResourcePlanDesc.java
Removing ql/src/test/queries/clientpositive/resourceplan.q
Removing ql/src/test/results/clientpositive/llap/resourceplan.q.out
Removing standalone-metastore/src/gen/org/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 2c871e4 HIVE-17139: Conditional expressions optimization: skip 
the expression evaluation if the condition is not satisfied for vectorization 
engine. (Jia Ke, reviewed by Ferdinand Xu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-10-12 02:06:40.352
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java:1
error: 
metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java: patch 
does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12891497 - PreCommit-HIVE-Build

> Move RawStore to standalone metastore
> -
>
> Key: HIVE-17733
> URL: https://issues.apache.org/jira/browse/HIVE-17733
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17733.2.patch, HIVE-17733.3.patch, 
> HIVE-17733.4.patch, HIVE-17733.patch
>
>
> This includes moving implementations of RawStore (like ObjectStore), 
> MetastoreDirectSql, and stats related classes like ColumnStatsAggregator and 
> the NDV classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17771) Implement create and show resource plan.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201332#comment-16201332
 ] 

Hive QA commented on HIVE-17771:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891434/HIVE-17771.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 11212 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
 (batchId=281)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7241/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7241/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7241/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12891434 - PreCommit-HIVE-Build

> Implement create and show resource plan.
> 
>
> Key: HIVE-17771
> URL: https://issues.apache.org/jira/browse/HIVE-17771
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-17771.01.patch
>
>
> Please see parent jira about llap workload management.
> This jira is to implement create and show resource plan commands in hive to 
> configure resource plans for llap workload.
> The following are the proposed commands implemented as part of the jira:
> CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism;
> SHOW RESOURCE PLAN;
> It will be followed up with more jiras to add pools, triggers and copy 
> resource plans. And also with drop commands for each of them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.

[Ferdinand Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferdinand Xu updated HIVE-17139:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Push to the master. Thanks [~Jk_Self] for the patch.

> Conditional expressions optimization: skip the expression evaluation if the 
> condition is not satisfied for vectorization engine.
> 
>
> Key: HIVE-17139
> URL: https://issues.apache.org/jira/browse/HIVE-17139
> Project: Hive
>  Issue Type: Improvement
>Reporter: Ke Jia
>Assignee: Ke Jia
> Fix For: 3.0.0
>
> Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, 
> HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, 
> HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, 
> HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, 
> HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, 
> HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, 
> HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, 
> HIVE-17139.8.patch, HIVE-17139.9.patch
>
>
> The case when and if statement execution for Hive vectorization is not 
> optimal, which all the conditional and else expressions are evaluated for 
> current implementation. The optimized approach is to update the selected 
> array of batch parameter after the conditional expression is executed. Then 
> the else expression will only do the selected rows instead of all.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17749) Multiple class have missed the ASF header

[Saijin Huang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201319#comment-16201319
 ] 

Saijin Huang commented on HIVE-17749:
-

[~lirui],can you plz take a quick review?

> Multiple class have missed the ASF header
> -
>
> Key: HIVE-17749
> URL: https://issues.apache.org/jira/browse/HIVE-17749
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Minor
> Attachments: HIVE-17749.1.patch
>
>
> Multiple class have missed the ASF header



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17646) MetaStoreUtils.isToInsertOnlyTable(Map<String, String> props) is not needed

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201320#comment-16201320
 ] 

Sergey Shelukhin commented on HIVE-17646:
-

This method is not used for validation IIRC, it's used to decide on conversion 
logic. One of them might now be redundant.

> MetaStoreUtils.isToInsertOnlyTable(Map props) is not needed
> ---
>
> Key: HIVE-17646
> URL: https://issues.apache.org/jira/browse/HIVE-17646
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>
> TransactionValidationListener is where all the logic to verify
> "transactional" & "transactional_properties" should be



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17646) MetaStoreUtils.isToInsertOnlyTable(Map<String, String> props) is not needed

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201307#comment-16201307
 ] 

Eugene Koifman commented on HIVE-17646:
---

TransactionValidationListener is called on every change to the the metastore 
data no matter how that change is made (e.g. HCat) so the validation is always 
performed

> MetaStoreUtils.isToInsertOnlyTable(Map props) is not needed
> ---
>
> Key: HIVE-17646
> URL: https://issues.apache.org/jira/browse/HIVE-17646
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>
> TransactionValidationListener is where all the logic to verify
> "transactional" & "transactional_properties" should be



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql

[ZhangBing Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201293#comment-16201293
 ] 

ZhangBing Lin commented on HIVE-17630:
--

[~dmtolpeko],Where do you find a problem? Or what parameters I need special 
settings?

> RESIGNAL:actual results are inconsistent with expectations at hplsql
> 
>
> Key: HIVE-17630
> URL: https://issues.apache.org/jira/browse/HIVE-17630
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.2.0, 3.0.0
>Reporter: ZhangBing Lin
>Assignee: Dmitry Tolpeko
>Priority: Minor
>
> when I execute example 3 at [http://www.hplsql.org/resignal]:
> BEGIN
>   DECLARE EXIT HANDLER FOR SQLEXCEPTION
>   BEGIN
> GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT;
> PRINT 'SQLSTATE: ' || SQLSTATE;
> PRINT 'Text: ' || text;
>   END; 
>  
>   BEGIN
> DECLARE EXIT HANDLER FOR SQLEXCEPTION
>   RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error';
>  
> SELECT * FROM abc.abc;-- Table does not exist, raise an exception
>   END;
> END;
> Actual results:
> SQLSTATE: 42S02
> Text: Error while compiling statement: FAILED: SemanticException [Error 
> 10001]: Line 1:14 Table not found 'abc'
>  
> The official result is：
> SQLSTATE: 02031
> Text: Some error



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17617) Rollup of an empty resultset should contain the grouping of the empty grouping set

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201290#comment-16201290
 ] 

Hive QA commented on HIVE-17617:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891440/HIVE-17617.05.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 11213 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby2] 
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby] 
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_cube1] 
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_cube_multi_gby] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_id3] 
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets1] 
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets3] 
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets4] 
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets5] 
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets_grouping]
 (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets_limit]
 (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_window] 
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup1] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_grouping_operators]
 (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[limit_pushdown2] 
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_grouping_sets] 
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[view_cbo] (batchId=66)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] 
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_cube1]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id1]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id2]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_id3]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets1]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets3]
 (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets4]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets5]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping]
 (batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_window]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_rollup1]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_grouping_sets]
 (batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_cube1] 
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_rollup1] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown2] 
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query18] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query22] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query27] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query36] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query5] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query67] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query70] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query77] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query80] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query86] 
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 

[jira] [Updated] (HIVE-17730) Queries can be closed automatically

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-17730:
--
Attachment: (was: HIVE-17730.01.patch)

> Queries can be closed automatically
> ---
>
> Key: HIVE-17730
> URL: https://issues.apache.org/jira/browse/HIVE-17730
> Project: Hive
>  Issue Type: Bug
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17730.03.patch
>
>
> HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
> manually and not by using try-with-resource. And now Query itself is auto 
> closeable, so we don't need the wrapper at all.
> So we should get rid of QueryWrapper and use try-with-resource to create 
> queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17730) Queries can be closed automatically

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-17730:
--
Status: Patch Available  (was: Open)

> Queries can be closed automatically
> ---
>
> Key: HIVE-17730
> URL: https://issues.apache.org/jira/browse/HIVE-17730
> Project: Hive
>  Issue Type: Bug
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17730.03.patch
>
>
> HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
> manually and not by using try-with-resource. And now Query itself is auto 
> closeable, so we don't need the wrapper at all.
> So we should get rid of QueryWrapper and use try-with-resource to create 
> queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17730) Queries can be closed automatically

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-17730:
--
Attachment: (was: HIVE-17730.02.patch)

> Queries can be closed automatically
> ---
>
> Key: HIVE-17730
> URL: https://issues.apache.org/jira/browse/HIVE-17730
> Project: Hive
>  Issue Type: Bug
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17730.03.patch
>
>
> HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
> manually and not by using try-with-resource. And now Query itself is auto 
> closeable, so we don't need the wrapper at all.
> So we should get rid of QueryWrapper and use try-with-resource to create 
> queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17730) Queries can be closed automatically

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-17730:
--
Attachment: HIVE-17730.03.patch

> Queries can be closed automatically
> ---
>
> Key: HIVE-17730
> URL: https://issues.apache.org/jira/browse/HIVE-17730
> Project: Hive
>  Issue Type: Bug
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17730.03.patch
>
>
> HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
> manually and not by using try-with-resource. And now Query itself is auto 
> closeable, so we don't need the wrapper at all.
> So we should get rid of QueryWrapper and use try-with-resource to create 
> queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17730) Queries can be closed automatically

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-17730:
--
Status: Open  (was: Patch Available)

> Queries can be closed automatically
> ---
>
> Key: HIVE-17730
> URL: https://issues.apache.org/jira/browse/HIVE-17730
> Project: Hive
>  Issue Type: Bug
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17730.01.patch, HIVE-17730.02.patch
>
>
> HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
> manually and not by using try-with-resource. And now Query itself is auto 
> closeable, so we don't need the wrapper at all.
> So we should get rid of QueryWrapper and use try-with-resource to create 
> queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-9995) ACID compaction tries to compact a single file

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-9995:
-
Attachment: HIVE-9995.WIP.patch

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17782) Inconsistent cast behavior from string to numeric types with regards to leading/trailing spaces

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere reassigned HIVE-17782:
-


> Inconsistent cast behavior from string to numeric types with regards to 
> leading/trailing spaces
> ---
>
> Key: HIVE-17782
> URL: https://issues.apache.org/jira/browse/HIVE-17782
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
>
> {noformat}
> select cast(' 1 ' as tinyint), cast(' 1 ' as smallint), cast(' 1 ' as int), 
> cast(' 1 ' as bigint), cast(' 1 ' as float), cast(' 1 ' as double), cast(' 1 
> ' as decimal(10,2))
> NULLNULLNULLNULL1.0 1.0 1
> {noformat}
> Looks like integer types (short, int, etc) fail the conversion due to the 
> leading/trailing spaces and return NULL, while float/double/decimal do not. 
> In fact, Decimal used to also return NULL in previous versions up until 
> HIVE-10799.
> Let's try to make this behavior consistent across all of these types, should 
> be simple enough to strip spaces before passing to number formatter.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17778) Add support for custom counters in trigger expression

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17778:
-
Attachment: HIVE-17778.1.patch

> Add support for custom counters in trigger expression
> -
>
> Key: HIVE-17778
> URL: https://issues.apache.org/jira/browse/HIVE-17778
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17778.1.patch
>
>
> HIVE-17508 only supports limited counters. This ticket is to extend it to 
> support custom counters (counters that are not supported by execution engine 
> will be dropped).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17781) Map MR settings to Tez settings via DeprecatedKeys

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-17781:

Status: Patch Available  (was: Open)

> Map MR settings to Tez settings via DeprecatedKeys
> --
>
> Key: HIVE-17781
> URL: https://issues.apache.org/jira/browse/HIVE-17781
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration, Tez
>Affects Versions: 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17781.1.patch
>
>
> Here's one that [~cdrome] and [~thiruvel] worked on:
> We found that certain Hadoop Map/Reduce settings that are set in site config 
> files do not take effect in Hive jobs, because the Tez site configs do not 
> contain the same settings.
> In Yahoo's case, the problem was that, at the time, there was no mapping 
> between {{MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART}} and 
> {{TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION}}. There were situations where 
> significant capacity on production clusters were being used up doing nothing, 
> while waiting for slow tasks to complete. This would have been avoided, were 
> the mappings in place.
> Tez provides a {{DeprecatedKeys}} utility class, to help map MR settings to 
> Tez settings. Hive should use this to ensure that the mappings are in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17781) Map MR settings to Tez settings via DeprecatedKeys

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-17781:

Attachment: HIVE-17781.1.patch

Submitting for tests.

> Map MR settings to Tez settings via DeprecatedKeys
> --
>
> Key: HIVE-17781
> URL: https://issues.apache.org/jira/browse/HIVE-17781
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration, Tez
>Affects Versions: 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17781.1.patch
>
>
> Here's one that [~cdrome] and [~thiruvel] worked on:
> We found that certain Hadoop Map/Reduce settings that are set in site config 
> files do not take effect in Hive jobs, because the Tez site configs do not 
> contain the same settings.
> In Yahoo's case, the problem was that, at the time, there was no mapping 
> between {{MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART}} and 
> {{TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION}}. There were situations where 
> significant capacity on production clusters were being used up doing nothing, 
> while waiting for slow tasks to complete. This would have been avoided, were 
> the mappings in place.
> Tez provides a {{DeprecatedKeys}} utility class, to help map MR settings to 
> Tez settings. Hive should use this to ensure that the mappings are in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17669) Cache to optimize SearchArgument deserialization

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201253#comment-16201253
 ] 

Mithun Radhakrishnan commented on HIVE-17669:
-

Thanks for pointing this out, [~prasanth_j]. I can commit this to 
{{branch-2.3}} as well.

> Cache to optimize SearchArgument deserialization
> 
>
> Key: HIVE-17669
> URL: https://issues.apache.org/jira/browse/HIVE-17669
> Project: Hive
>  Issue Type: Improvement
>  Components: ORC, Query Processor
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17669.3.patch, HIVE-17669.4.patch, 
> HIVE-17699.1.patch, HIVE-17699.2.patch
>
>
> And another, from [~selinazh] and [~cdrome]. (YHIVE-927)
> When a mapper needs to process multiple ORC files, it might land up having 
> use essentially the same {{SearchArgument}} over several files. It would be 
> good not to have to deserialize from string, over and over again. Caching the 
> object against the string-form should speed things up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17534) Add a config to turn off parquet vectorization

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201247#comment-16201247
 ] 

Vihang Karajgaonkar commented on HIVE-17534:


Thanks [~mmccline] for taking a look. Here is the review board link 
https://reviews.apache.org/r/62915/

> Add a config to turn off parquet vectorization
> --
>
> Key: HIVE-17534
> URL: https://issues.apache.org/jira/browse/HIVE-17534
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-17534.01.patch, HIVE-17534.02.patch
>
>
> It should be a good addition to give an option for users to turn off parquet 
> vectorization without affecting vectorization on other file formats. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17781) Map MR settings to Tez settings via DeprecatedKeys

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan reassigned HIVE-17781:
---


> Map MR settings to Tez settings via DeprecatedKeys
> --
>
> Key: HIVE-17781
> URL: https://issues.apache.org/jira/browse/HIVE-17781
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration, Tez
>Affects Versions: 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
>
> Here's one that [~cdrome] and [~thiruvel] worked on:
> We found that certain Hadoop Map/Reduce settings that are set in site config 
> files do not take effect in Hive jobs, because the Tez site configs do not 
> contain the same settings.
> In Yahoo's case, the problem was that, at the time, there was no mapping 
> between {{MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART}} and 
> {{TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION}}. There were situations where 
> significant capacity on production clusters were being used up doing nothing, 
> while waiting for slow tasks to complete. This would have been avoided, were 
> the mappings in place.
> Tez provides a {{DeprecatedKeys}} utility class, to help map MR settings to 
> Tez settings. Hive should use this to ensure that the mappings are in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17780) MM table followup list

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17780:
--
Issue Type: Task  (was: Bug)

> MM table followup list
> --
>
> Key: HIVE-17780
> URL: https://issues.apache.org/jira/browse/HIVE-17780
> Project: Hive
>  Issue Type: Task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> List of files that need more attention:
> itests/src/test/resources/testconfiguration.properties
> ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java - 
> ok??
> ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java ?
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java - ?
> ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java - ?!
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java - !?!
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java - ?
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java - look for usages of 
> isFullAcidTable().  Directory.aborted? Used by Compactor; OK
> ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java - ??
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java - ?
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractBucketJoinProc.java - 
> disables bucket join for MM
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java - ?!
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingOpProcFactory.java
>  - filed jira
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java - ?
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java - come back to 
> this
> ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 
>  -? mostly logs, but 1 change?
> ql/src/test/queries/clientnegative/mm_bucket_convert.q - cannot convert 
> bucketed MM tale to non-MM (
> ql/src/test/queries/clientnegative/mm_concatenate.q - disables Concate for MM
> ql/src/test/queries/clientnegative/mm_truncate_cols.q - trunctate columns not 
> supported
> ql/src/test/queries/clientpositive/mm_all.q
> ql/src/test/queries/clientpositive/mm_buckets.q - test TABLESAMPLE statement
> ql/src/test/queries/clientpositive/mm_conversions.q - make MM table regular 
> table (I don't think we should to this)
> ql/src/test/queries/clientpositive/mm_exchangepartition.q
> ql/src/test/queries/clientpositive/mm_exim.q - does this respect aborted txns?
> ql/src/test/queries/clientpositive/mm_loaddata.q
> standalone-metastore/src/main/thrift/hive_metastore.thrift
> also see HIVE-17691 and HIVE-14535



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17534) Add a config to turn off parquet vectorization

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201205#comment-16201205
 ] 

Matt McCline commented on HIVE-17534:
-

[~vihang] Can you create a review board.  There is enough change here it is 
hard to follow as a patch file.  And, please include [~sershe] as a reviewer.  
Thanks.

> Add a config to turn off parquet vectorization
> --
>
> Key: HIVE-17534
> URL: https://issues.apache.org/jira/browse/HIVE-17534
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-17534.01.patch, HIVE-17534.02.patch
>
>
> It should be a good addition to give an option for users to turn off parquet 
> vectorization without affecting vectorization on other file formats. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17780) MM table followup list

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-17780:
-

Assignee: Eugene Koifman

> MM table followup list
> --
>
> Key: HIVE-17780
> URL: https://issues.apache.org/jira/browse/HIVE-17780
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> List of files that need more attention:
> itests/src/test/resources/testconfiguration.properties
> ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java - 
> ok??
> ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java ?
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java - ?
> ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java - ?!
> ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java - !?!
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java - ?
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java - look for usages of 
> isFullAcidTable().  Directory.aborted? Used by Compactor; OK
> ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java - ??
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java - ?
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractBucketJoinProc.java - 
> disables bucket join for MM
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java - ?!
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/BucketingSortingOpProcFactory.java
>  - filed jira
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java - ?
> ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java - come back to 
> this
> ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 
>  -? mostly logs, but 1 change?
> ql/src/test/queries/clientnegative/mm_bucket_convert.q - cannot convert 
> bucketed MM tale to non-MM (
> ql/src/test/queries/clientnegative/mm_concatenate.q - disables Concate for MM
> ql/src/test/queries/clientnegative/mm_truncate_cols.q - trunctate columns not 
> supported
> ql/src/test/queries/clientpositive/mm_all.q
> ql/src/test/queries/clientpositive/mm_buckets.q - test TABLESAMPLE statement
> ql/src/test/queries/clientpositive/mm_conversions.q - make MM table regular 
> table (I don't think we should to this)
> ql/src/test/queries/clientpositive/mm_exchangepartition.q
> ql/src/test/queries/clientpositive/mm_exim.q - does this respect aborted txns?
> ql/src/test/queries/clientpositive/mm_loaddata.q
> standalone-metastore/src/main/thrift/hive_metastore.thrift
> also see HIVE-17691 and HIVE-14535



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201196#comment-16201196
 ] 

Hive QA commented on HIVE-17139:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890998/HIVE-17139.20.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 11211 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=239)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish
 (batchId=186)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7239/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7239/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7239/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890998 - PreCommit-HIVE-Build

> Conditional expressions optimization: skip the expression evaluation if the 
> condition is not satisfied for vectorization engine.
> 
>
> Key: HIVE-17139
> URL: https://issues.apache.org/jira/browse/HIVE-17139
> Project: Hive
>  Issue Type: Improvement
>Reporter: Ke Jia
>Assignee: Ke Jia
> Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, 
> HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, 
> HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, 
> HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, 
> HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, 
> HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, 
> HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, 
> HIVE-17139.8.patch, HIVE-17139.9.patch
>
>
> The case when and if statement execution for Hive vectorization is not 
> optimal, which all the conditional and else expressions are evaluated for 
> current implementation. The optimized approach is to update the selected 
> array of batch parameter after the conditional expression is executed. Then 
> the else expression will only do the selected rows instead of all.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HIVE-17779) Add itests profile

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan resolved HIVE-17779.
-
Resolution: Invalid

This profile already exists.

> Add itests profile
> --
>
> Key: HIVE-17779
> URL: https://issues.apache.org/jira/browse/HIVE-17779
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Reporter: Vijay Kumar
>Assignee: Vijay Kumar
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17691) Miscellaneous List

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17691:
--
Description: 
# DDLSemanticAnalyzer.alterTableOutput is unused
# DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
TransactionManager
# DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
place..
# FileSinkOperator has multiple places that look like _conf.getWriteType() == 
AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() != 
AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
MoveTask.handleStaticParts() call to Hive.loadPartition()
# HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
obsolete
# Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
either because DbTxnManager.acquireLocks() does  
_compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
non-acid tables
# In general integration with full Acid seems confused wrt to MM and seems to 
treat MM as special table type rather than subtype of Acid table.  (mostly, but 
not always).
## e.g. _SemanticAnalyzer.genBucketingSortingDest(String dest, Operator input, 
QB qb, TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx)_ 
##  _SemanticAnalyzer.validate()_ has _if (tbl != null && 
(AcidUtils.isFullAcidTable(tbl) || 
MetaStoreUtils.isInsertOnlyTable(tbl.getParameters( {_
# LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
than from TM
# ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
Need to verify we properly commit the txn in the Driver
# As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
MR.  This doesn't exercise some code specifically for dealing with writes from 
Union All queries (CTAS, Insert into).  On MR this requires 
"hive.optimize.union.remove=true" (false by default)
# Remove MoveWork().setNoop(boolean) and usages per todo in 
_GenMapRedUtils.createMRWorkForMergingFiles (FileSinkOperator fsInput, Path 
finalName, DependencyCollectionTask dependencyTask,   List 
mvTasks, HiveConf conf,   Task currTask)_
# PartialScanWork.tblDesc - unused
# _Partition.getBucketPath(int bucketNum)_ has "// Note: this makes assumptions 
that won't work with MM tables, unions, etc.".  File Jira?
# _PartitionDesc.LOG_ is unused
# Insert Overwrite for MM is incomplete - see comments in HIVE-15212 regarding 
IOW and multi IOW
# mm_bucket_convert.q - doesn't install DbTxnManager, doesn't write any data - 
not sure what it tests in practice
# There no concurrency tests that check locking
# no tests with aborted txns
# tests don't run on Tez/LLap - affects some optimization like Union All writes





  was:
# DDLSemanticAnalyzer.alterTableOutput is unused
# DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
TransactionManager
# DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
place..
# FileSinkOperator has multiple places that look like _conf.getWriteType() == 
AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() != 
AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
MoveTask.handleStaticParts() call to Hive.loadPartition()
# HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
obsolete
# Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
either because DbTxnManager.acquireLocks() does  
_compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
non-acid tables
# In general integration with full Acid seems confused wrt to MM and seems to 
treat MM as special table type rather than subtype of Acid table.  (mostly, but 
not always).
## e.g. _SemanticAnalyzer.genBucketingSortingDest(String dest, Operator input, 
QB qb, TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx)_ 
##  _SemanticAnalyzer.validate()_ has _if (tbl != null && 
(AcidUtils.isFullAcidTable(tbl) || 
MetaStoreUtils.isInsertOnlyTable(tbl.getParameters( {_
# LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
than from TM
# ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
Need to verify we properly commit the txn in the Driver
# As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
MR.  This doesn't exercise some code specifically for dealing with writes from 
Union All 

[jira] [Commented] (HIVE-17669) Cache to optimize SearchArgument deserialization

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201174#comment-16201174
 ] 

Prasanth Jayachandran commented on HIVE-17669:
--

[~mithun] I did not pull this into branch-2.2 as this is perf optimization and 
not bug fix. Do you want to revert the patch from branch-2.2 or commit the 
patch to branch-2.3 as well to sync the branches? I am fine with either. 

> Cache to optimize SearchArgument deserialization
> 
>
> Key: HIVE-17669
> URL: https://issues.apache.org/jira/browse/HIVE-17669
> Project: Hive
>  Issue Type: Improvement
>  Components: ORC, Query Processor
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17669.3.patch, HIVE-17669.4.patch, 
> HIVE-17699.1.patch, HIVE-17699.2.patch
>
>
> And another, from [~selinazh] and [~cdrome]. (YHIVE-927)
> When a mapper needs to process multiple ORC files, it might land up having 
> use essentially the same {{SearchArgument}} over several files. It would be 
> good not to have to deserialize from string, over and over again. Caching the 
> object against the string-form should speed things up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15212) merge branch into master

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201171#comment-16201171
 ] 

Sergey Shelukhin commented on HIVE-15212:
-

+1 also from my side. The branch contains a lot of patches from other people 
that I was reviewing over time.

> merge branch into master
> 
>
> Key: HIVE-15212
> URL: https://issues.apache.org/jira/browse/HIVE-15212
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, 
> HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, 
> HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, 
> HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, 
> HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, 
> HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch, 
> HIVE-15212.16.patch, HIVE-15212.17.patch, HIVE-15212.18.patch, 
> HIVE-15212.19.patch, HIVE-15212.20.patch, HIVE-15212.21.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17765) expose Hive keywords

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201173#comment-16201173
 ] 

Thejas M Nair commented on HIVE-17765:
--

Javadoc for 
https://docs.oracle.com/javase/7/docs/api/java/sql/DatabaseMetaData.html#getSQLKeywords()
bq. Retrieves a comma-separated list of all of this database's SQL keywords 
that are NOT also SQL:2003 keywords.

ODBC behavior seems to be similar, but different -
bq. SQL_KEYWORDSstring  A string of all the keywords at the database 
management system that are not in the ODBC's list of reserved words.
(from 
https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/odbc/src/tpc/db2z_fngetinfo.html
 )

However, this patch seems to be returning all the keywords.

> expose Hive keywords 
> -
>
> Key: HIVE-17765
> URL: https://issues.apache.org/jira/browse/HIVE-17765
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17765.nogen.patch, HIVE-17765.patch
>
>
> This could be useful e.g. for BI tools (via ODBC/JDBC drivers) to decide on 
> SQL capabilities of Hive



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17779) Add itests profile

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan reassigned HIVE-17779:
---


> Add itests profile
> --
>
> Key: HIVE-17779
> URL: https://issues.apache.org/jira/browse/HIVE-17779
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Reporter: Vijay Kumar
>Assignee: Vijay Kumar
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15212) merge branch into master

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201170#comment-16201170
 ] 

Gunther Hagleitner commented on HIVE-15212:
---

+1

> merge branch into master
> 
>
> Key: HIVE-15212
> URL: https://issues.apache.org/jira/browse/HIVE-15212
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, 
> HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, 
> HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, 
> HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, 
> HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, 
> HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch, 
> HIVE-15212.16.patch, HIVE-15212.17.patch, HIVE-15212.18.patch, 
> HIVE-15212.19.patch, HIVE-15212.20.patch, HIVE-15212.21.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17669) Cache to optimize SearchArgument deserialization

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201155#comment-16201155
 ] 

Mithun Radhakrishnan commented on HIVE-17669:
-

Updated documentation, 
[here|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties].

> Cache to optimize SearchArgument deserialization
> 
>
> Key: HIVE-17669
> URL: https://issues.apache.org/jira/browse/HIVE-17669
> Project: Hive
>  Issue Type: Improvement
>  Components: ORC, Query Processor
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17669.3.patch, HIVE-17669.4.patch, 
> HIVE-17699.1.patch, HIVE-17699.2.patch
>
>
> And another, from [~selinazh] and [~cdrome]. (YHIVE-927)
> When a mapper needs to process multiple ORC files, it might land up having 
> use essentially the same {{SearchArgument}} over several files. It would be 
> good not to have to deserialize from string, over and over again. Caching the 
> object against the string-form should speed things up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17669) Cache to optimize SearchArgument deserialization

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201143#comment-16201143
 ] 

Mithun Radhakrishnan commented on HIVE-17669:
-

Thanks for the review, [~prasanth_j]. I've picked this into {{branch-2.2}} as 
well. Cheers!

> Cache to optimize SearchArgument deserialization
> 
>
> Key: HIVE-17669
> URL: https://issues.apache.org/jira/browse/HIVE-17669
> Project: Hive
>  Issue Type: Improvement
>  Components: ORC, Query Processor
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17669.3.patch, HIVE-17669.4.patch, 
> HIVE-17699.1.patch, HIVE-17699.2.patch
>
>
> And another, from [~selinazh] and [~cdrome]. (YHIVE-927)
> When a mapper needs to process multiple ORC files, it might land up having 
> use essentially the same {{SearchArgument}} over several files. It would be 
> good not to have to deserialize from string, over and over again. Caching the 
> object against the string-form should speed things up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17777) Add maven coordinates in itests/pom.xml

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-1:

Attachment: HIVE-1.patch

Patch from [~vijay_k]

> Add maven coordinates in itests/pom.xml
> ---
>
> Key: HIVE-1
> URL: https://issues.apache.org/jira/browse/HIVE-1
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure, Testing Infrastructure
>Reporter: Vijay Kumar
>Assignee: Vijay Kumar
> Attachments: HIVE-1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17777) Add maven coordinates in itests/pom.xml

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-1:

Status: Patch Available  (was: Open)

> Add maven coordinates in itests/pom.xml
> ---
>
> Key: HIVE-1
> URL: https://issues.apache.org/jira/browse/HIVE-1
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure, Testing Infrastructure
>Reporter: Vijay Kumar
>Assignee: Vijay Kumar
> Attachments: HIVE-1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15212) merge branch into master

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201136#comment-16201136
 ] 

Eugene Koifman commented on HIVE-15212:
---

+1

> merge branch into master
> 
>
> Key: HIVE-15212
> URL: https://issues.apache.org/jira/browse/HIVE-15212
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, 
> HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, 
> HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, 
> HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, 
> HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, 
> HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch, 
> HIVE-15212.16.patch, HIVE-15212.17.patch, HIVE-15212.18.patch, 
> HIVE-15212.19.patch, HIVE-15212.20.patch, HIVE-15212.21.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16395) ConcurrentModificationException on config object in HoS

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-16395:
--
Status: Patch Available  (was: Open)

> ConcurrentModificationException on config object in HoS
> ---
>
> Key: HIVE-16395
> URL: https://issues.apache.org/jira/browse/HIVE-16395
> Project: Hive
>  Issue Type: Task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Andrew Sherman
> Attachments: HIVE-16395.1.patch
>
>
> Looks like this is happening inside spark executors, looks to be some race 
> condition when modifying {{Configuration}} objects.
> Stack-Trace:
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:267)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:213)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:334)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:682)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:240)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>   at org.apache.spark.scheduler.Task.run(Task.scala:89)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:253)
>   ... 21 more
> Caused by: java.util.ConcurrentModificationException
>   at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
>   at 
> org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2455)
>   at 
> org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:716)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:181)
>   at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815)
>   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
>   at 
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852)
>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834)
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
>   at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
>   at 
> org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:108)
>   at 
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:68)
>   ... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-16395) ConcurrentModificationException on config object in HoS

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman reassigned HIVE-16395:
-

Assignee: Andrew Sherman  (was: Sahil Takiar)

> ConcurrentModificationException on config object in HoS
> ---
>
> Key: HIVE-16395
> URL: https://issues.apache.org/jira/browse/HIVE-16395
> Project: Hive
>  Issue Type: Task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Andrew Sherman
> Attachments: HIVE-16395.1.patch
>
>
> Looks like this is happening inside spark executors, looks to be some race 
> condition when modifying {{Configuration}} objects.
> Stack-Trace:
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:267)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:213)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:334)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:682)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:240)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>   at org.apache.spark.scheduler.Task.run(Task.scala:89)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:253)
>   ... 21 more
> Caused by: java.util.ConcurrentModificationException
>   at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
>   at 
> org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2455)
>   at 
> org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:716)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:181)
>   at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815)
>   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
>   at 
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852)
>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834)
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
>   at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
>   at 
> org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:108)
>   at 
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:68)
>   ... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16395) ConcurrentModificationException on config object in HoS

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-16395:
--
Attachment: HIVE-16395.1.patch

> ConcurrentModificationException on config object in HoS
> ---
>
> Key: HIVE-16395
> URL: https://issues.apache.org/jira/browse/HIVE-16395
> Project: Hive
>  Issue Type: Task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Andrew Sherman
> Attachments: HIVE-16395.1.patch
>
>
> Looks like this is happening inside spark executors, looks to be some race 
> condition when modifying {{Configuration}} objects.
> Stack-Trace:
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>   at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:267)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:213)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:334)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:682)
>   at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:240)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
>   at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
>   at 
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>   at org.apache.spark.scheduler.Task.run(Task.scala:89)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:253)
>   ... 21 more
> Caused by: java.util.ConcurrentModificationException
>   at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
>   at 
> org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2455)
>   at 
> org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:716)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:181)
>   at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815)
>   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
>   at 
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852)
>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834)
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
>   at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
>   at 
> org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:108)
>   at 
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
>   at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:68)
>   ... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17635) Add unit tests to CompactionTxnHandler and use PreparedStatements for queries

[Andrew Sherman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201122#comment-16201122
 ] 

Andrew Sherman commented on HIVE-17635:
---

There is already a [review board review 
here|https://reviews.apache.org/r/62693/]
There should be no logic changes.

> Add unit tests to CompactionTxnHandler and use PreparedStatements for queries
> -
>
> Key: HIVE-17635
> URL: https://issues.apache.org/jira/browse/HIVE-17635
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-17635.1.patch
>
>
> It is better for jdbc code that runs against the HMS database to use 
> PreparedStatements. Convert CompactionTxnHandler queries to use 
> PreparedStatement and add tests to TestCompactionTxnHandler to test these 
> queries, and improve code coverage.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17777) Add maven coordinates in itests/pom.xml

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan reassigned HIVE-1:
---


> Add maven coordinates in itests/pom.xml
> ---
>
> Key: HIVE-1
> URL: https://issues.apache.org/jira/browse/HIVE-1
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure, Testing Infrastructure
>Reporter: Vijay Kumar
>Assignee: Vijay Kumar
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17778) Add support for custom counters in trigger expression

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-17778:



> Add support for custom counters in trigger expression
> -
>
> Key: HIVE-17778
> URL: https://issues.apache.org/jira/browse/HIVE-17778
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> HIVE-17508 only supports limited counters. This ticket is to extend it to 
> support custom counters (counters that are not supported by execution engine 
> will be dropped).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201115#comment-16201115
 ] 

Aihua Xu commented on HIVE-15016:
-

Reattach patch-4 with netty-all dependency added to llap-server project. When I 
built the whole thing, it passed, but I could see the compilation error when I 
compiled the individual project.

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-15016:

Status: Patch Available  (was: Open)

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-15016:

Status: Open  (was: Patch Available)

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-15016:

Attachment: (was: HIVE-15016.4.patch)

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-15016:

Attachment: HIVE-15016.4.patch

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201086#comment-16201086
 ] 

Sergey Shelukhin edited comment on HIVE-12727 at 10/11/17 10:32 PM:


The legacy mode is actually more restrictive than the new mode. The cartesian 
product check used to be true in the strict mode (the default), and false, with 
all the other checks, in nonstrict.
The new configs only apply in new versions; if you set hive.mapred.mode to null 
on older versions, it assumes the default value of "strict", applying all the 
checks.
Otherwise please describe the scenario (the old value, old version of Hive, new 
version of Hive, and what changes were made to this setting) that you are 
encountering, afaik there should be no upgrade scenarios  where this patch 
changes the behavior in a manner that breaks the previous user-set 
configuration, or makes the checks more restrictive if the configuration was 
not set.


was (Author: sershe):
The legacy mode is actually more restrictive than the new mode. The cartesian 
product check used to be true in the strict mode (the default), and false, with 
all the other checks, in nonstrict.
The new configs only apply in new versions; if you set hive.mapred.mode to null 
on older versions, it assumes the default value of "strict", applying all the 
checks.
Otherwise please describe the scenario (the old value, old version of Hive, new 
version of Hive, and what changes were made to this setting) that you are 
encountering, afaik it should have no upgrade ases where it changes the 
behavior in a manner that breaks the previous user-specific setting, or makes 
the checks more restrictive if the setting was not set.

> refactor Hive strict checks to be more granular, allow order by no limit and 
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
>  Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, 
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, 
> HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal 
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while 
> compiling statement: FAILED: SemanticException [Error 10041]: No partition 
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default 
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, 
> and strict, for backward compat for people who are relying on strict already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201086#comment-16201086
 ] 

Sergey Shelukhin commented on HIVE-12727:
-

The legacy mode is actually more restrictive than the new mode. The cartesian 
product check used to be true in the strict mode (the default), and false, with 
all the other checks, in nonstrict.
The new configs only apply in new versions; if you set hive.mapred.mode to null 
on older versions, it assumes the default value of "strict", applying all the 
checks.
Otherwise please describe the scenario (the old value, old version of Hive, new 
version of Hive, and what changes were made to this setting) that you are 
encountering, afaik it should have no upgrade ases where it changes the 
behavior in a manner that breaks the previous user-specific setting, or makes 
the checks more restrictive if the setting was not set.

> refactor Hive strict checks to be more granular, allow order by no limit and 
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
>  Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, 
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, 
> HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal 
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while 
> compiling statement: FAILED: SemanticException [Error 10041]: No partition 
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default 
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, 
> and strict, for backward compat for people who are relying on strict already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17635) Add unit tests to CompactionTxnHandler and use PreparedStatements for queries

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201077#comment-16201077
 ] 

Sahil Takiar commented on HIVE-17635:
-

[~asherman] can you create an RB for this? There seem to be some logic changes 
in addition to the migration to {{PreparedStatement}} - whats the nature of 
these changes?

> Add unit tests to CompactionTxnHandler and use PreparedStatements for queries
> -
>
> Key: HIVE-17635
> URL: https://issues.apache.org/jira/browse/HIVE-17635
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-17635.1.patch
>
>
> It is better for jdbc code that runs against the HMS database to use 
> PreparedStatements. Convert CompactionTxnHandler queries to use 
> PreparedStatement and add tests to TestCompactionTxnHandler to test these 
> queries, and improve code coverage.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17730) Queries can be closed automatically

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201073#comment-16201073
 ] 

Hive QA commented on HIVE-17730:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891556/HIVE-17730.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 978 failed/errored test(s), 5600 tests 
executed
*Failed tests:*
{noformat}
TestAvroHCatLoader - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestDbTxnManager2 - did not produce a TEST-*.xml file (likely timed out) 
(batchId=283)
TestE2EScenarios - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestHCatLoaderComplexSchema - did not produce a TEST-*.xml file (likely timed 
out) (batchId=184)
TestHCatLoaderEncryption - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestHCatLoaderStorer - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestHCatStorer - did not produce a TEST-*.xml file (likely timed out) 
(batchId=183)
TestHCatStorerMulti - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestHCatStorerWrapper - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestKeyWrapperFactory - did not produce a TEST-*.xml file (likely timed out) 
(batchId=270)
TestLog4j2Appenders - did not produce a TEST-*.xml file (likely timed out) 
(batchId=270)
TestOrcHCatLoader - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestParquetHCatLoader - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestParquetHCatStorer - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestPigHCatUtil - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestRCFileHCatLoader - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestRemoteHiveMetaStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=209)
TestSequenceFileHCatStorer - did not produce a TEST-*.xml file (likely timed 
out) (batchId=184)
TestSetUGIOnBothClientServer - did not produce a TEST-*.xml file (likely timed 
out) (batchId=206)
TestSetUGIOnOnlyClient - did not produce a TEST-*.xml file (likely timed out) 
(batchId=204)
TestSetUGIOnOnlyServer - did not produce a TEST-*.xml file (likely timed out) 
(batchId=214)
TestTextFileHCatLoader - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestTextFileHCatStorer - did not produce a TEST-*.xml file (likely timed out) 
(batchId=184)
TestTxnCommands - did not produce a TEST-*.xml file (likely timed out) 
(batchId=282)
TestTxnNoBuckets - did not produce a TEST-*.xml file (likely timed out) 
(batchId=270)
TestUtil - did not produce a TEST-*.xml file (likely timed out) (batchId=184)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key2]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index] 
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins] 
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBeeLineDriver.org.apache.hadoop.hive.cli.TestBeeLineDriver
 (batchId=239)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreCliDriver
 (batchId=242)
org.apache.hadoop.hive.cli.TestBlobstoreNegativeCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreNegativeCliDriver
 (batchId=242)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=18)

[jira] [Assigned] (HIVE-17776) Add CliDriver and tests for HDFS with Erasure Coding enabled

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar reassigned HIVE-17776:
---


> Add CliDriver and tests for HDFS with Erasure Coding enabled
> 
>
> Key: HIVE-17776
> URL: https://issues.apache.org/jira/browse/HIVE-17776
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Erasure Coding is a new feature in Hadoop 3.x and we should add some tests to 
> ensure Hive works properly with EC. We can do something similar to what we 
> did for HDFS encryption support.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201045#comment-16201045
 ] 

Aihua Xu commented on HIVE-15016:
-

That is strange. Let me try to clean up and rebuild.

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement global execution triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.11.patch

Fixes some NPE causing test failures.

> Implement global execution triggers based on counters
> -
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.10.patch, 
> HIVE-17508.11.patch, HIVE-17508.2.patch, HIVE-17508.3.patch, 
> HIVE-17508.3.patch, HIVE-17508.4.patch, HIVE-17508.5.patch, 
> HIVE-17508.6.patch, HIVE-17508.7.patch, HIVE-17508.8.patch, 
> HIVE-17508.9.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Triggers that are bound to a resource plan. 
> Each trigger can have a trigger expression and an action associated with it. 
> Trigger expressions are evaluated at runtime after configurable check 
> interval, based on which actions like killing a query, moving a query to 
> different pool etc. will get invoked. Simple execution trigger could be 
> something like
> {code}
> CREATE TRIGGER slow_query IN global
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17669) Cache to optimize SearchArgument deserialization

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17669:
-
   Resolution: Fixed
Fix Version/s: 2.4.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Committed to branch-2 and master. Thanks [~mithun] for the patch!

> Cache to optimize SearchArgument deserialization
> 
>
> Key: HIVE-17669
> URL: https://issues.apache.org/jira/browse/HIVE-17669
> Project: Hive
>  Issue Type: Improvement
>  Components: ORC, Query Processor
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17669.3.patch, HIVE-17669.4.patch, 
> HIVE-17699.1.patch, HIVE-17699.2.patch
>
>
> And another, from [~selinazh] and [~cdrome]. (YHIVE-927)
> When a mapper needs to process multiple ORC files, it might land up having 
> use essentially the same {{SearchArgument}} over several files. It would be 
> good not to have to deserialize from string, over and over again. Caching the 
> object against the string-form should speed things up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17669) Cache to optimize SearchArgument deserialization

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200990#comment-16200990
 ] 

Prasanth Jayachandran commented on HIVE-17669:
--

+1. Will commit this shortly.

> Cache to optimize SearchArgument deserialization
> 
>
> Key: HIVE-17669
> URL: https://issues.apache.org/jira/browse/HIVE-17669
> Project: Hive
>  Issue Type: Improvement
>  Components: ORC, Query Processor
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-17669.3.patch, HIVE-17669.4.patch, 
> HIVE-17699.1.patch, HIVE-17699.2.patch
>
>
> And another, from [~selinazh] and [~cdrome]. (YHIVE-927)
> When a mapper needs to process multiple ORC files, it might land up having 
> use essentially the same {{SearchArgument}} over several files. It would be 
> good not to have to deserialize from string, over and over again. Caching the 
> object against the string-form should speed things up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200988#comment-16200988
 ] 

Ashutosh Chauhan commented on HIVE-15016:
-

With latest .4 patch I am still getting failure locally:
{code}
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 2:15.137s
[INFO] Finished at: Wed Oct 11 14:24:34 PDT 2017
[INFO] Final Memory: 192M/1369M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) 
on project hive-llap-server: Compilation failure: Compilation failure:
[ERROR] 
/Users/ashutosh/workspace/apache-master/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java:[122,21]
 package io.netty.util does not exist
[ERROR] 
/Users/ashutosh/workspace/apache-master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/SystemConfigurationServlet.java:[40,21]
 package io.netty.util does not exist
[ERROR] -> [Help 1]
{code}

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200951#comment-16200951
 ] 

Aihua Xu commented on HIVE-15016:
-

[~ashutoshc] That seems to be caused by later commit after I submitted the 
patch. I already rebased and resubmitted for a new build. Waiting for the new 
build. The build is passing for me locally.

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17726) Using exists may lead to incorrect results

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200941#comment-16200941
 ] 

Hive QA commented on HIVE-17726:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891415/HIVE-17726.2.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11211 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_createas1] 
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16] 
(batchId=241)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94] 
(batchId=241)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16] 
(batchId=239)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94] 
(batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow 
(batchId=240)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection 
(batchId=240)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=240)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg 
(batchId=240)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth 
(batchId=240)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth 
(batchId=240)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth 
(batchId=240)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth 
(batchId=240)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7237/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7237/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7237/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12891415 - PreCommit-HIVE-Build

> Using exists may lead to incorrect results
> --
>
> Key: HIVE-17726
> URL: https://issues.apache.org/jira/browse/HIVE-17726
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Zoltan Haindrich
>Assignee: Vineet Garg
> Attachments: HIVE-17726.1.patch, HIVE-17726.2.patch
>
>
> {code}
> drop table if exists tx1;
> create table tx1 (a integer,b integer);
> insert into tx1   values  (1, 1),
> (1, 2),
> (1, 3);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b limit 1);
> {code}
> current results are 6 and 2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200934#comment-16200934
 ] 

Ashutosh Chauhan commented on HIVE-15016:
-

[~aihuaxu] Latest patch is still failing with:
{code}
[ERROR] 
/data/hiveptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java:[122,21]
 package io.netty.util does not exist
[ERROR] 
/data/hiveptest/working/apache-github-source-source/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/SystemConfigurationServlet.java:[40,21]
 package io.netty.util does not exist
{code}

> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, 
> HIVE-15016.4.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17775) PTF should take advantage of partitioning, bucketing and sorting

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17775:

Description: 
If a PTF partitions-by a partition column of the table (or a bucketing column, 
although that could be more difficult), or orders-by a partition column or a 
column on which the table is already sorted being bucketed+sorted, it should 
take advantage of these properties of the table.
That can turn some PTF operators (that I'm presently observing on a real query 
from a user) into a single scan, or at least a much simpler DAG; as it is, the 
query is very slow.

  was:
When the user is running a PTF that partitions by a partition column of the 
table (or a bucketing column, although that could be more difficult), or orders 
by a partition column, or a column on which the table is already sorted being 
bucketed+sorted, the PTF should take advantage of these.
That can turn some PTF operators (that I'm presently observing on a real query 
from a user) into a single scan, or at least a much simpler DAG; as it is, the 
query is very slow.


> PTF should take advantage of partitioning, bucketing and sorting
> 
>
> Key: HIVE-17775
> URL: https://issues.apache.org/jira/browse/HIVE-17775
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> If a PTF partitions-by a partition column of the table (or a bucketing 
> column, although that could be more difficult), or orders-by a partition 
> column or a column on which the table is already sorted being 
> bucketed+sorted, it should take advantage of these properties of the table.
> That can turn some PTF operators (that I'm presently observing on a real 
> query from a user) into a single scan, or at least a much simpler DAG; as it 
> is, the query is very slow.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HIVE-17391) Compaction fails if there is an empty value in tblproperties

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200917#comment-16200917
 ] 

Steve Yeom edited comment on HIVE-17391 at 10/11/17 8:37 PM:
-

Talked with Ashutosh regarding this jira. Got great inputs.

I think we can differentiate the following cases in StringableMap.java's 
methods. 
Specifically its toString() and its constructors when it loops on Map.Entry's 
to process each entry. 
1. key is not null nor empty string, value is null.
2. key is not null nor empty string, value is empty string.

Another factor is Properties which is inherited from Hashtable which does 
not allow null key nor value (vs StringableMap which is inherited from 
HashMap).
So when StringableMap is converting itself to Properties object, we gets to the 
NullPointerException problem
of this jira simply Hashtable is supposed to throw a NullPointerException when 
a caller tries to add 
null key/value. 


was (Author: steveyeom2017):
Talked with Ashutosh regarding this jira.

I think we can differentiate the following cases in StringableMap.java's 
methods. 
Specifically its toString() and its constructors when it loops on Map.Entry's 
to process each entry. 
1. key is not null nor empty string, value is null.
2. key is not null nor empty string, value is empty string.

Another factor is Properties which is inherited from Hashtable which does 
not allow null key nor value.
So when StringableMap is converting itself to Properties object, we gets to the 
NullPointerException problem
of this jira simply Hashtable is supposed to throw a NullPointerException when 
a caller tries to add 
null key/value. 

> Compaction fails if there is an empty value in tblproperties
> 
>
> Key: HIVE-17391
> URL: https://issues.apache.org/jira/browse/HIVE-17391
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Transactions
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Ashutosh Chauhan
>Assignee: Steve Yeom
> Attachments: HIVE-17391.01.patch
>
>
> create table t1 (a int) tblproperties ('serialization.null.format'='');
> alter table t1 compact 'major';
> fails



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17391) Compaction fails if there is an empty value in tblproperties

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200917#comment-16200917
 ] 

Steve Yeom commented on HIVE-17391:
---

Talked with Ashutosh regarding this jira.

I think we can differentiate the following cases in StringableMap.java's 
methods. 
Specifically its toString() and its constructors when it loops on Map.Entry's 
to process each entry. 
1. key is not null nor empty string, value is null.
2. key is not null nor empty string, value is empty string.

Another factor is Properties which is inherited from Hashtable which does 
not allow null key nor value.
So when StringableMap is converting itself to Properties object, we gets to the 
NullPointerException problem
of this jira simply Hashtable is supposed to throw a NullPointerException when 
a caller tries to add 
null key/value. 

> Compaction fails if there is an empty value in tblproperties
> 
>
> Key: HIVE-17391
> URL: https://issues.apache.org/jira/browse/HIVE-17391
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Transactions
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Ashutosh Chauhan
>Assignee: Steve Yeom
> Attachments: HIVE-17391.01.patch
>
>
> create table t1 (a int) tblproperties ('serialization.null.format'='');
> alter table t1 compact 'major';
> fails



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17775) PTF should take advantage of partitioning, bucketing and sorting

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200915#comment-16200915
 ] 

Sergey Shelukhin commented on HIVE-17775:
-

cc [~jcamachorodriguez] [~ashutoshc]

> PTF should take advantage of partitioning, bucketing and sorting
> 
>
> Key: HIVE-17775
> URL: https://issues.apache.org/jira/browse/HIVE-17775
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> When the user is running a PTF that partitions by a partition column of the 
> table (or a bucketing column, although that could be more difficult), or 
> orders by a partition column, or a column on which the table is already 
> sorted being bucketed+sorted, the PTF should take advantage of these.
> That can turn some PTF operators (that I'm presently observing on a real 
> query from a user) into a single scan, or at least a much simpler DAG; as it 
> is, the query is very slow.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17691) Miscellaneous List

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17691:
--
Description: 
# DDLSemanticAnalyzer.alterTableOutput is unused
# DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
TransactionManager
# DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
place..
# FileSinkOperator has multiple places that look like _conf.getWriteType() == 
AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() != 
AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
MoveTask.handleStaticParts() call to Hive.loadPartition()
# HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
obsolete
# Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
either because DbTxnManager.acquireLocks() does  
_compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
non-acid tables
# In general integration with full Acid seems confused wrt to MM and seems to 
treat MM as special table type rather than subtype of Acid table.  (mostly, but 
not always).
## e.g. _SemanticAnalyzer.genBucketingSortingDest(String dest, Operator input, 
QB qb, TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx)_ 
##  _SemanticAnalyzer.validate()_ has _if (tbl != null && 
(AcidUtils.isFullAcidTable(tbl) || 
MetaStoreUtils.isInsertOnlyTable(tbl.getParameters( {_
# LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
than from TM
# ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
Need to verify we properly commit the txn in the Driver
# As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
MR.  This doesn't exercise some code specifically for dealing with writes from 
Union All queries (CTAS, Insert into).  On MR this requires 
"hive.optimize.union.remove=true" (false by default)
# Remove MoveWork().setNoop(boolean) and usages per todo in 
_GenMapRedUtils.createMRWorkForMergingFiles (FileSinkOperator fsInput, Path 
finalName, DependencyCollectionTask dependencyTask,   List 
mvTasks, HiveConf conf,   Task currTask)_
# PartialScanWork.tblDesc - unused
# _Partition.getBucketPath(int bucketNum)_ has "// Note: this makes assumptions 
that won't work with MM tables, unions, etc.".  File Jira?
# _PartitionDesc.LOG_ is unused
# Insert Overwrite for MM is incomplete - see comments in HIVE-15212 regarding 
IOW and multi IOW




  was:
# DDLSemanticAnalyzer.alterTableOutput is unused
# DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
TransactionManager
# DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
place..
# FileSinkOperator has multiple places that look like _conf.getWriteType() == 
AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() != 
AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
MoveTask.handleStaticParts() call to Hive.loadPartition()
# HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
obsolete
# Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
either because DbTxnManager.acquireLocks() does  
_compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
non-acid tables
# In general integration with full Acid seems confused wrt to MM and seems to 
treat MM as special table type rather than subtype of Acid table.  (mostly, but 
not always).
## e.g. _SemanticAnalyzer.genBucketingSortingDest(String dest, Operator input, 
QB qb, TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx)_ 
# LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
than from TM
# ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
Need to verify we properly commit the txn in the Driver
# As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
MR.  This doesn't exercise some code specifically for dealing with writes from 
Union All queries (CTAS, Insert into).  On MR this requires 
"hive.optimize.union.remove=true" (false by default)
# Remove MoveWork().setNoop(boolean) and usages per todo in 
_GenMapRedUtils.createMRWorkForMergingFiles (FileSinkOperator fsInput, Path 
finalName, DependencyCollectionTask dependencyTask,   List 
mvTasks, HiveConf conf,   Task currTask)_
# PartialScanWork.tblDesc - unused
# _Partition.getBucketPath(int bucketNum)_ has 

[jira] [Updated] (HIVE-17691) Miscellaneous List

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17691:
--
Description: 
# DDLSemanticAnalyzer.alterTableOutput is unused
# DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
TransactionManager
# DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
place..
# FileSinkOperator has multiple places that look like _conf.getWriteType() == 
AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() != 
AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
MoveTask.handleStaticParts() call to Hive.loadPartition()
# HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
obsolete
# Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
either because DbTxnManager.acquireLocks() does  
_compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
non-acid tables
# In general integration with full Acid seems confused wrt to MM and seems to 
treat MM as special table type rather than subtype of Acid table.  (mostly, but 
not always).
## e.g. _SemanticAnalyzer.genBucketingSortingDest(String dest, Operator input, 
QB qb, TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx)_ 
# LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
than from TM
# ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
Need to verify we properly commit the txn in the Driver
# As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
MR.  This doesn't exercise some code specifically for dealing with writes from 
Union All queries (CTAS, Insert into).  On MR this requires 
"hive.optimize.union.remove=true" (false by default)
# Remove MoveWork().setNoop(boolean) and usages per todo in 
_GenMapRedUtils.createMRWorkForMergingFiles (FileSinkOperator fsInput, Path 
finalName, DependencyCollectionTask dependencyTask,   List 
mvTasks, HiveConf conf,   Task currTask)_
# PartialScanWork.tblDesc - unused
# _Partition.getBucketPath(int bucketNum)_ has "// Note: this makes assumptions 
that won't work with MM tables, unions, etc.".  File Jira?
# _PartitionDesc.LOG_ is unused
# Insert Overwrite for MM is incomplete - see comments in HIVE-15212 regarding 
IOW and multi IOW




  was:
# DDLSemanticAnalyzer.alterTableOutput is unused
# DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
TransactionManager
# DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
place..
# FileSinkOperator has multiple places that look like _conf.getWriteType() == 
AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() != 
AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
MoveTask.handleStaticParts() call to Hive.loadPartition()
# HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
obsolete
# Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
either because DbTxnManager.acquireLocks() does  
_compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
non-acid tables
# In general integration with full Acid seems confused wrt to MM and seems to 
treat MM as special table type rather than subtype of Acid table.  (mostly, but 
not always).
## e.g. _SemanticAnalyzer.genBucketingSortingDest(String dest, Operator input, 
QB qb, TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx)_ 
# LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
than from TM
# ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
Need to verify we properly commit the txn in the Driver
# As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
MR.  This doesn't exercise some code specifically for dealing with writes from 
Union All queries (CTAS, Insert into).  On MR this requires 
"hive.optimize.union.remove=true" (false by default)
# Remove MoveWork().setNoop(boolean) and usages per todo in 
_GenMapRedUtils.createMRWorkForMergingFiles (FileSinkOperator fsInput, Path 
finalName, DependencyCollectionTask dependencyTask,   List 
mvTasks, HiveConf conf,   Task currTask)_
# PartialScanWork.tblDesc - unused
# _Partition.getBucketPath(int bucketNum)_ has "// Note: this makes assumptions 
that won't work with MM tables, unions, etc.".  File Jira?
# _PartitionDesc.LOG_ is unused
# 





> Miscellaneous List

[jira] [Commented] (HIVE-17774) compaction may start with 0 splits and fail

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200878#comment-16200878
 ] 

Eugene Koifman commented on HIVE-17774:
---

yes, the fix for this is in CompactorMR
from
{noformat}
if (parsedDeltas.size() == 0 && dir.getOriginalFiles() == null) {
{noformat}
to 
{noformat}
if (parsedDeltas.size() == 0 && dir.getOriginalFiles().size() == 0) {
{noformat}


> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17774) compaction may start with 0 splits and fail

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200872#comment-16200872
 ] 

Sergey Shelukhin commented on HIVE-17774:
-

[~ekoifman] is it possible to fix for branch-2, branch-1 etc? that patch is too 
big to backport there.

> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17774) compaction may start with 0 splits and fail

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200869#comment-16200869
 ] 

Eugene Koifman commented on HIVE-17774:
---

this is fixed in HIVE-14988

> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17774) compaction may start with 0 splits and fail

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17774:
--
Component/s: Transactions

> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17691) Miscellaneous List

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17691:
--
Description: 
# DDLSemanticAnalyzer.alterTableOutput is unused
# DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
TransactionManager
# DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
place..
# FileSinkOperator has multiple places that look like _conf.getWriteType() == 
AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() != 
AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
MoveTask.handleStaticParts() call to Hive.loadPartition()
# HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
obsolete
# Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
either because DbTxnManager.acquireLocks() does  
_compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
non-acid tables
# In general integration with full Acid seems confused wrt to MM and seems to 
treat MM as special table type rather than subtype of Acid table.  (mostly, but 
not always).
## e.g. _SemanticAnalyzer.genBucketingSortingDest(String dest, Operator input, 
QB qb, TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx)_ 
# LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
than from TM
# ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
Need to verify we properly commit the txn in the Driver
# As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
MR.  This doesn't exercise some code specifically for dealing with writes from 
Union All queries (CTAS, Insert into).  On MR this requires 
"hive.optimize.union.remove=true" (false by default)
# Remove MoveWork().setNoop(boolean) and usages per todo in 
_GenMapRedUtils.createMRWorkForMergingFiles (FileSinkOperator fsInput, Path 
finalName, DependencyCollectionTask dependencyTask,   List 
mvTasks, HiveConf conf,   Task currTask)_
# PartialScanWork.tblDesc - unused
# _Partition.getBucketPath(int bucketNum)_ has "// Note: this makes assumptions 
that won't work with MM tables, unions, etc.".  File Jira?
# _PartitionDesc.LOG_ is unused
# 




  was:
# DDLSemanticAnalyzer.alterTableOutput is unused
# DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
TransactionManager
# DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
place..
# FileSinkOperator has multiple places that look like _conf.getWriteType() == 
AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() != 
AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
MoveTask.handleStaticParts() call to Hive.loadPartition()
# HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
obsolete
# Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
either because DbTxnManager.acquireLocks() does  
_compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
non-acid tables
# In general integration with full Acid seems confused wrt to MM and seems to 
treat MM as special table type rather than subtype of Acid table.  (mostly, but 
not always).
# LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
than from TM
# ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
Need to verify we properly commit the txn in the Driver
# As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
MR.  This doesn't exercise some code specifically for dealing with writes from 
Union All queries (CTAS, Insert into).  On MR this requires 
"hive.optimize.union.remove=true" (false by default)
# Remove MoveWork().setNoop(boolean) and usages per todo in 
_GenMapRedUtils.createMRWorkForMergingFiles (FileSinkOperator fsInput, Path 
finalName, DependencyCollectionTask dependencyTask,   List 
mvTasks, HiveConf conf,   Task currTask)_
# PartialScanWork.tblDesc - unused
# _Partition.getBucketPath(int bucketNum)_ has "// Note: this makes assumptions 
that won't work with MM tables, unions, etc.".  File Jira?
# _PartitionDesc.LOG_ is unused
# 





> Miscellaneous List
> --
>
> Key: HIVE-17691
> URL: https://issues.apache.org/jira/browse/HIVE-17691
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>

[jira] [Updated] (HIVE-17464) Fix to be able to disable max shuffle size DHJ config

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-17464:
---
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master, thanks [~ashutoshc]!

> Fix to be able to disable max shuffle size DHJ config
> -
>
> Key: HIVE-17464
> URL: https://issues.apache.org/jira/browse/HIVE-17464
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 3.0.0
>
> Attachments: HIVE-17464.patch
>
>
> Setting {{hive.auto.convert.join.shuffle.max.size}} to -1 does not work as 
> expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17774) compaction may start with 0 splits and fail

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-17774:
---


> compaction may start with 0 splits and fail
> ---
>
> Key: HIVE-17774
> URL: https://issues.apache.org/jira/browse/HIVE-17774
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>
> {noformat}
> 2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR 
> (CompactorMR.java:launchCompactionJob(295)) - 
> Submitting MINOR compaction job 
>  (current delta dirs count=0, obsolete delta dirs count=0. 
> TxnIdRange[9223372036854775807,-9223372036854775808]
> ...
> 2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter 
> (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
> ...
> 2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job 
> (Job.java:monitorAndPrintJob(1380)) - 
> Job job_1503950256860_15982 failed with state FAILED due to: No of maps and 
> reduces are 0 job_1503950256860_15982
> Job commit failed: java.io.FileNotFoundException: File 
> .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 
> does not exist.
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
>   at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
>   at 
> org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
>   at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Looks like the MR job should not have been attempted in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16827) Merge stats task and column stats task into a single task

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-16827:

Attachment: HIVE-16827.05wip02.patch

> Merge stats task and column stats task into a single task
> -
>
> Key: HIVE-16827
> URL: https://issues.apache.org/jira/browse/HIVE-16827
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Zoltan Haindrich
> Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch, 
> HIVE-16827.03.patch, HIVE-16827.04wip01.patch, HIVE-16827.04wip02.patch, 
> HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch, HIVE-16827.04wip05.patch, 
> HIVE-16827.04wip06.patch, HIVE-16827.04wip07.patch, HIVE-16827.04wip08.patch, 
> HIVE-16827.04wip09.patch, HIVE-16827.04wip10.patch, HIVE-16827.05wip01.patch, 
> HIVE-16827.05wip02.patch, HIVE-16827.4.patch
>
>
> Within the task, we can specify whether to compute basic stats only or column 
> stats only or both.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17773) add test to make sure bucket pruning doesn't clash with unbucketed acid tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-17773:
-


> add test to make sure bucket pruning doesn't clash with unbucketed acid tables
> --
>
> Key: HIVE-17773
> URL: https://issues.apache.org/jira/browse/HIVE-17773
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> HIVE-11525 add bucket pruning.  Unbucketed acid tables name data files in the 
> same way as proper/enforced buckets.  
> Bucket pruning should only kick in for bucketed tables 
> (https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java#L109)
>   but it's good to add some tests to make sure



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17730) Queries can be closed automatically

[Alexander Kolbasov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200825#comment-16200825
 ] 

Alexander Kolbasov commented on HIVE-17730:
---

There are several cases in the code:

1) Users of existing QueryWrapper
2) Code that uses Query, but cals rollbackAndCleanup to close it
3) Code that calls (or doesn't) query.closeAll()

This patch converts *all* uses of Query in ObjectStore to auto-closeable use 
that guarantees that it is closed always closed.
Note that after collection query close we can't call size() on it, so in such 
cases query is copied to another collection.

As part of the patch I also cleaned up various small things that don't warrant 
individual JIRAs.

> Queries can be closed automatically
> ---
>
> Key: HIVE-17730
> URL: https://issues.apache.org/jira/browse/HIVE-17730
> Project: Hive
>  Issue Type: Bug
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17730.01.patch, HIVE-17730.02.patch
>
>
> HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
> manually and not by using try-with-resource. And now Query itself is auto 
> closeable, so we don't need the wrapper at all.
> So we should get rid of QueryWrapper and use try-with-resource to create 
> queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17730) Queries can be closed automatically

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated HIVE-17730:
--
Attachment: HIVE-17730.02.patch

> Queries can be closed automatically
> ---
>
> Key: HIVE-17730
> URL: https://issues.apache.org/jira/browse/HIVE-17730
> Project: Hive
>  Issue Type: Bug
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Attachments: HIVE-17730.01.patch, HIVE-17730.02.patch
>
>
> HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
> manually and not by using try-with-resource. And now Query itself is auto 
> closeable, so we don't need the wrapper at all.
> So we should get rid of QueryWrapper and use try-with-resource to create 
> queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-15267) Make query length calculation logic more accurate in TxnUtils.needNewQuery()

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200788#comment-16200788
 ] 

Hive QA commented on HIVE-15267:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891528/HIVE-15267.04.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11211 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join5]
 (batchId=173)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=101)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=239)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7236/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7236/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7236/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12891528 - PreCommit-HIVE-Build

> Make query length calculation logic more accurate in TxnUtils.needNewQuery()
> 
>
> Key: HIVE-15267
> URL: https://issues.apache.org/jira/browse/HIVE-15267
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Transactions
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Wei Zheng
>Assignee: Steve Yeom
> Attachments: HIVE-15267.01.patch, HIVE-15267.02.patch, 
> HIVE-15267.03.patch, HIVE-15267.04.patch
>
>
> In HIVE-15181 there's such review comment, for which this ticket will handle
> {code}
> in TxnUtils.needNewQuery() "sizeInBytes / 1024 > queryMemoryLimit" doesn't do 
> the right thing.
> If the user sets METASTORE_DIRECT_SQL_MAX_QUERY_LENGTH to 1K, they most 
> likely want each SQL string to be at most 1K.
> But if sizeInBytes=2047, this still returns false.
> It should include length of "suffix" in computation of sizeInBytes
> Along the same lines: the check for max query length is done after each batch 
> is already added to the query. Suppose there are 1000 9-digit txn IDs in each 
> IN(...). That's, conservatively, 18KB of text. So the length of each query is 
> increasing in 18KB chunks. 
> I think the check for query length should be done for each item in IN clause.
> If some DB has a limit on query length of X, then any query > X will fail. So 
> I think this must ensure not to produce any queries > X, even by 1 char.
> For example, case 3.1 of the UT generates a query of almost 4000 characters - 
> this is clearly > 1KB.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17760) Create a unit test which validates HIVE-9423 does not regress

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-17760:
--
Attachment: HIVE-17760.2.patch

> Create a unit test which validates HIVE-9423 does not regress 
> --
>
> Key: HIVE-17760
> URL: https://issues.apache.org/jira/browse/HIVE-17760
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-17760.1.patch, HIVE-17760.2.patch
>
>
> During [HIVE-9423] we verified that when the Thrift server pool is exhausted, 
> then Beeline connection times out, and provide a meaningful error message.
> Create a unit test which verifies this, and helps to keep this feature working



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17464) Fix to be able to disable max shuffle size DHJ config

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200725#comment-16200725
 ] 

Ashutosh Chauhan commented on HIVE-17464:
-

+1 : )

> Fix to be able to disable max shuffle size DHJ config
> -
>
> Key: HIVE-17464
> URL: https://issues.apache.org/jira/browse/HIVE-17464
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17464.patch
>
>
> Setting {{hive.auto.convert.join.shuffle.max.size}} to -1 does not work as 
> expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17464) Fix to be able to disable max shuffle size DHJ config

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200719#comment-16200719
 ] 

Jesus Camacho Rodriguez commented on HIVE-17464:


Yes, it is missing the review :)

> Fix to be able to disable max shuffle size DHJ config
> -
>
> Key: HIVE-17464
> URL: https://issues.apache.org/jira/browse/HIVE-17464
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17464.patch
>
>
> Setting {{hive.auto.convert.join.shuffle.max.size}} to -1 does not work as 
> expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17464) Fix to be able to disable max shuffle size DHJ config

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200711#comment-16200711
 ] 

Ashutosh Chauhan commented on HIVE-17464:
-

[~jcamachorodriguez] This shall get in?

> Fix to be able to disable max shuffle size DHJ config
> -
>
> Key: HIVE-17464
> URL: https://issues.apache.org/jira/browse/HIVE-17464
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17464.patch
>
>
> Setting {{hive.auto.convert.join.shuffle.max.size}} to -1 does not work as 
> expected.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17708) Upgrade surefire to 2.20.1

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-17708:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks, Zoltan!

> Upgrade surefire to 2.20.1
> --
>
> Key: HIVE-17708
> URL: https://issues.apache.org/jira/browse/HIVE-17708
> Project: Hive
>  Issue Type: Sub-task
>  Components: Build Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Fix For: 3.0.0
>
> Attachments: HIVE-17708.01.patch
>
>
> with the current 2.18.1 jdk9 test execution result in:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on 
> project hive-common: Execution default-test of goal 
> org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test failed: 
> java.lang.NoClassDefFoundError: java/sql/Timestamp: java.sql.Timestamp -> 
> [Help 1]
> {code}
> make sure that it works *reliably* (2.19.1 had a bug which made idea debuging 
> problematic)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17726) Using exists may lead to incorrect results

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200667#comment-16200667
 ] 

Ashutosh Chauhan commented on HIVE-17726:
-

+1 

> Using exists may lead to incorrect results
> --
>
> Key: HIVE-17726
> URL: https://issues.apache.org/jira/browse/HIVE-17726
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Zoltan Haindrich
>Assignee: Vineet Garg
> Attachments: HIVE-17726.1.patch, HIVE-17726.2.patch
>
>
> {code}
> drop table if exists tx1;
> create table tx1 (a integer,b integer);
> insert into tx1   values  (1, 1),
> (1, 2),
> (1, 3);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b);
> select count(*) as result,3 as expected from tx1 u
> where exists (select * from tx1 v where u.a=v.a and u.b <> v.b limit 1);
> {code}
> current results are 6 and 2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17534) Add a config to turn off parquet vectorization

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200665#comment-16200665
 ] 

Vihang Karajgaonkar commented on HIVE-17534:


[~mmccline] [~Ferd] Can you please take a look?

> Add a config to turn off parquet vectorization
> --
>
> Key: HIVE-17534
> URL: https://issues.apache.org/jira/browse/HIVE-17534
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-17534.01.patch, HIVE-17534.02.patch
>
>
> It should be a good addition to give an option for users to turn off parquet 
> vectorization without affecting vectorization on other file formats. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17508) Implement global execution triggers based on counters

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200646#comment-16200646
 ] 

Hive QA commented on HIVE-17508:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891395/HIVE-17508.10.patch

{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 41 failed/errored test(s), 10637 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
 (batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
 (batchId=100)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
 (batchId=101)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query39] 
(batchId=239)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion 
(batchId=216)
org.apache.hadoop.hive.ql.TestAcidOnTez.testBucketedAcidInsertWithRemoveUnion 
(batchId=216)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=216)
org.apache.hadoop.hive.ql.TestAcidOnTez.testInsertWithRemoveUnion (batchId=216)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=216)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMergeJoinOnTez (batchId=216)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=216)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=222)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressWithHiveServer2ProgressBarDisabled
 (batchId=222)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
 (batchId=227)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithExecutionMode
 (batchId=224)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithNoneMode
 (batchId=224)
org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithTez.testFetchResultsOfLogWithPerformanceMode
 (batchId=224)

[jira] [Updated] (HIVE-17534) Add a config to turn off parquet vectorization

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-17534:
---
Attachment: HIVE-17534.02.patch

Spark test was not running possibly due to the missing .q in the 
TestConfiguration.properties. Fixed the LlapDecider.q failure. Attaching second 
version of the patch.

> Add a config to turn off parquet vectorization
> --
>
> Key: HIVE-17534
> URL: https://issues.apache.org/jira/browse/HIVE-17534
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-17534.01.patch, HIVE-17534.02.patch
>
>
> It should be a good addition to give an option for users to turn off parquet 
> vectorization without affecting vectorization on other file formats. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)