[jira] [Commented] (HIVE-18113) Remove mixed partitions/table schema support

[Zoltan Haindrich (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262114#comment-16262114
 ] 

Zoltan Haindrich commented on HIVE-18113:
-

[~ashutoshc]: my intention was to change the "cascade" option into a 
permanently enabled feature - so the only thing changed is that a partitioned 
table's *all* partition agrees on the actual schema.

I think that employing views to describe the schema changes could achieve the 
same goal with better transparency for the user ; and could even add extra 
flexibility (invoke sql functions/etc).


> Remove mixed partitions/table schema support
> 
>
> Key: HIVE-18113
> URL: https://issues.apache.org/jira/browse/HIVE-18113
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18113.01.patch
>
>
> Currently it is possible to have misaligned table/partition schemas;
> [see here for example| 
> https://github.com/apache/hive/blob/a5c2e15c7cc125d8cda2ee3a8ed64c116ff6b755/ql/src/test/queries/clientpositive/schema_evol_text_vec_part.q#L156]
> result of [this insert 
> statement|https://github.com/apache/hive/blob/a5c2e15c7cc125d8cda2ee3a8ed64c116ff6b755/ql/src/test/queries/clientpositive/schema_evol_text_vec_part.q#L162]
>  are these [null 
> values|https://github.com/apache/hive/blob/a5c2e15c7cc125d8cda2ee3a8ed64c116ff6b755/ql/src/test/results/clientpositive/llap/schema_evol_text_vec_part.q.out#L660]
> This mixed partition setup can cause the stats aggregation to become quite 
> confusing...I think it would be better to remove this thingthere is a 
> {{CASCADE}} flag already; which changes the schema all over the 
> table/partitions/etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17972) Implement Parquet vectorization reader for Map type

[Colin Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262108#comment-16262108
 ] 

Colin Ma commented on HIVE-17972:
-

[~Ferd], the patch is uploaded, can you help to review, thanks.

> Implement Parquet vectorization reader for Map type
> ---
>
> Key: HIVE-17972
> URL: https://issues.apache.org/jira/browse/HIVE-17972
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-17972.001.patch
>
>
> Parquet vectorized reader can't support map type, it should be supported to 
> improve the performance when the query with map type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17972) Implement Parquet vectorization reader for Map type

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated HIVE-17972:

Status: Patch Available  (was: Open)

> Implement Parquet vectorization reader for Map type
> ---
>
> Key: HIVE-17972
> URL: https://issues.apache.org/jira/browse/HIVE-17972
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-17972.001.patch
>
>
> Parquet vectorized reader can't support map type, it should be supported to 
> improve the performance when the query with map type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17972) Implement Parquet vectorization reader for Map type

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated HIVE-17972:

Attachment: HIVE-17972.001.patch

> Implement Parquet vectorization reader for Map type
> ---
>
> Key: HIVE-17972
> URL: https://issues.apache.org/jira/browse/HIVE-17972
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-17972.001.patch
>
>
> Parquet vectorized reader can't support map type, it should be supported to 
> improve the performance when the query with map type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18112) show create for view having special char in where clause is not showing properly

[Naresh P R (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262099#comment-16262099
 ] 

Naresh P R commented on HIVE-18112:
---

Thanks for the help [~ashutoshc]
>From current master branch code, i could see as part of HIVE-14013, along with 
>describe issue, show create issue is also fixed in 2.2.1 & higher versions.
I want this fix in 2.1.0 & 1.2.0


> show create for view having special char in where clause is not showing 
> properly
> 
>
> Key: HIVE-18112
> URL: https://issues.apache.org/jira/browse/HIVE-18112
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HIVE-18112.patch
>
>
> e.g., 
> CREATE VIEW `v2` AS select `evil_byte1`.`a` from `default`.`EVIL_BYTE1` where 
> `evil_byte1`.`a` = 'abcÖdefÖgh';
> Output:
> ==
> 0: jdbc:hive2://172.26.122.227:1> show create table v2;
> ++--+
> | createtab_stmt  
>|
> ++--+
> | CREATE VIEW `v2` AS select `evil_byte1`.`a` from `default`.`EVIL_BYTE1` 
> where `evil_byte1`.`a` = 'abc�def�gh'  |
> ++--+
> Only show create output is having invalid characters, actual source table 
> content is displayed properly in the console.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17856) MM tables - IOW is not ACID compliant

[Steve Yeom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yeom updated HIVE-17856:
--
Attachment: HIVE-17856.14.patch

> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
>  Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, 
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, 
> HIVE-17856.14.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, 
> HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, 
> HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should 
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas, 
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true", 
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm; 
> create table iow1_mm(key int) partitioned by (key2 int)  
> tblproperties("transactional"="true", 
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from 
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key 
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1, 
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as 
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties 
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18100) Some tests time out

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262092#comment-16262092
 ] 

Hive QA commented on HIVE-18100:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898788/HIVE-18100.3.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11410 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7955/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7955/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7955/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898788 - PreCommit-HIVE-Build

> Some tests time out
> ---
>
> Key: HIVE-18100
> URL: https://issues.apache.org/jira/browse/HIVE-18100
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-18100.1.patch, HIVE-18100.2.patch, 
> HIVE-18100.3.patch, HIVE-18100.patch
>
>
> Some tests had 100s of queries in a single query which times out resulting in 
> Hive QA failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18043) Vectorization: Support List type in MapWork

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated HIVE-18043:

Attachment: HIVE-18043.003.patch

Thanks [~Ferd] for review, update the patch according to your comments and add 
test cases target to ListIndexColColumn.

> Vectorization: Support List type in MapWork
> ---
>
> Key: HIVE-18043
> URL: https://issues.apache.org/jira/browse/HIVE-18043
> Project: Hive
>  Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18043.001.patch, HIVE-18043.002.patch, 
> HIVE-18043.003.patch
>
>
> Support Complex Types in vectorization is finished in HIVE-16589, but List 
> type is still not support in MapWork. It should be supported to improve the 
> performance when vectorization is enable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18129) The ConditionalResolverMergeFiles doesn't merge empty files

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li reassigned HIVE-18129:
-


> The ConditionalResolverMergeFiles doesn't merge empty files
> ---
>
> Key: HIVE-18129
> URL: https://issues.apache.org/jira/browse/HIVE-18129
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>
> If a query produces lots of empty files, these files won't be merged by the 
> merge-small-file feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17361:
--
Attachment: HIVE-17361.21.patch

> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, 
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, 
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, 
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, 
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch, 
> HIVE-17361.3.patch, HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap 
> between ACID table and regular hive table.
> Current Documentation is under [DML 
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
>  and [Loading files into 
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it 
> uses the input file name which may not be in 0_0 which can break some 
> read logic.  (Certainly will for Acid).
> * It does not check the schema of the file.  This may be a non issue for Acid 
> which requires ORC which is self describing so Schema Evolution may handle 
> this seamlessly.  (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible. 
> * Bucketed (and thus sorted) tables don't support Load Data (but only if 
> hive.strict.checks.bucketing=true (default)).  Will keep this restriction for 
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a 
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created before files are moved, delta_8_8 
> becomes visible before it's populated.  Is that an issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262033#comment-16262033
 ] 

Hive QA commented on HIVE-17361:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898783/HIVE-17361.20.patch

{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11418 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_data_into_acid]
 (batchId=91)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7954/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7954/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7954/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898783 - PreCommit-HIVE-Build

> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, 
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, 
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, 
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, 
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.3.patch, 
> HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap 
> between ACID table and regular hive table.
> Current Documentation is under [DML 
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
>  and [Loading files into 
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it 
> uses the input file name which may not be in 0_0 which can break some 
> read logic.  (Certainly will for Acid).
> * It does not check the schema of the file.  This may be a non issue for Acid 
> which requires ORC which is self describing so Schema Evolution may handle 
> this seamlessly.  (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible. 
> * Bucketed (and thus sorted) tables don't support Load Data (but only if 
> hive.strict.checks.bucketing=true (default)).  Will keep this restriction for 
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a 
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created before files are moved, delta_8_8 
> becomes visible before it's populated.  Is that an issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18111) Fix temp path for Spark DPP sink

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262016#comment-16262016
 ] 

Rui Li commented on HIVE-18111:
---

Operator IDs might not be unique, e.g. when we clone the operator tree, cloned 
operator will have the same ID as original operator. So the 3rd patch adds a 
unique ID to each DPP sink operator, and uses this unique ID as event source ID 
in map work.
[~stakiar], [~xuefuz] could you take a look? Thanks.

> Fix temp path for Spark DPP sink
> 
>
> Key: HIVE-18111
> URL: https://issues.apache.org/jira/browse/HIVE-18111
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-18111.1.patch, HIVE-18111.2.patch, 
> HIVE-18111.3.patch
>
>
> Before HIVE-17877, each DPP sink has only one target work. The output path of 
> a DPP work is {{TMP_PATH/targetWorkId/dppWorkId}}. When we do the pruning, 
> each map work reads DPP outputs under {{TMP_PATH/targetWorkId}}.
> After HIVE-17877, each DPP sink can have multiple target works. It's possible 
> that a map work needs to read DPP outputs from multiple 
> {{TMP_PATH/targetWorkId}}. To solve this, I think we can have a DPP output 
> path specific to each query, e.g. {{QUERY_TMP_PATH/dpp_output}}. Each DPP 
> work outputs to {{QUERY_TMP_PATH/dpp_output/dppWorkId}}. And each map work 
> reads from {{QUERY_TMP_PATH/dpp_output}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18090) acid heartbeat fails when metastore is connected via hadoop credential

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-18090:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Patch committed to master !

> acid heartbeat fails when metastore is connected via hadoop credential
> --
>
> Key: HIVE-18090
> URL: https://issues.apache.org/jira/browse/HIVE-18090
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Transactions
>Affects Versions: 1.3.0, 2.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-18090.0.patch
>
>
> steps to recreate the issue. assuming two users 
> * test
> * another 
> create two jceks files for each user and place them on hdfs with access to 
> that file only allowed to the user. hdfs locations with permissions 
> {code}
> -rwx--   1 another another492 2017-11-16 13:06 
> /user/another/another.jceks
> -rwx--   1 test test489 2017-11-16 13:05 /user/test/test.jceks
> {code}
> password used to create 
> * /user/another/another.jceks -- another
> * /user/test/test.jceks -- test
> on core-site.xml 
> {code}
> 
> hadoop.proxyuser.[superuser].hosts
> *
> 
> 
> hadoop.proxyuser.[superuser].groups
> *
> 
> {code}
> and restart hdfs.
> enable ACID on HS2 (change the required properties).additional changes on  
> hiveserver2 configs 
> {code}
> * hive.metastore.warehouse.dir=file:///tmp/hive/test-warehouse
> * hive.server2.enable.doAs=true
> * remove javax.jdo.option.ConnectionPassword property from hive-site.xml
> {code}
> start hiveserver2
> connect to the server using beeline using any user:
> {code}
> create table a (i int, b string);
> insert into a values (0 , '0'), (1 , '1'), (2 , '2'), (3 , '3'), (4 , '4'), 
> (5 , '5'), (6 , '6'), (7 , '7'), (8 , '8'), (9 , '9'), (10 , '10'), (11 , 
> '11'), (12 , '12'), (13 , '13'), (14 , '14'), (15 , '15'), (16 , '16'), (17 , 
> '17'), (18 , '18'), (19 , '19'), (20 , '20'), (21 , '21'), (22 , '22'), (23 , 
> '23'), (24 , '24'), (25 , '25'), (26 , '26'), (27 , '27'), (28 , '28'), (29 , 
> '29'), (30 , '30'), (31 , '31'), (32 , '32'), (33 , '33'), (34 , '34'), (35 , 
> '35'), (36 , '36'), (37 , '37'), (38 , '38'), (39 , '39'), (40 , '40'), (41 , 
> '41'), (42 , '42'), (43 , '43'), (44 , '44'), (45 , '45'), (46 , '46'), (47 , 
> '47'), (48 , '48'), (49 , '49'), (50 , '50'), (51 , '51'), (52 , '52'), (53 , 
> '53'), (54 , '54'), (55 , '55'), (56 , '56'), (57 , '57'), (58 , '58'), (59 , 
> '59'), (60 , '60'), (61 , '61'), (62 , '62'), (63 , '63'), (64 , '64'), (65 , 
> '65'), (66 , '66'), (67 , '67'), (68 , '68'), (69 , '69'), (70 , '70'), (71 , 
> '71'), (72 , '72'), (73 , '73'), (74 , '74'), (75 , '75'), (76 , '76'), (77 , 
> '77'), (78 , '78'), (79 , '79'), (80 , '80'), (81 , '81'), (82 , '82'), (83 , 
> '83'), (84 , '84'), (85 , '85'), (86 , '86'), (87 , '87'), (88 , '88'), (89 , 
> '89'), (90 , '90'), (91 , '91'), (92 , '92'), (93 , '93'), (94 , '94'), (95 , 
> '95'), (96 , '96'), (97 , '97'), (98 , '98'), (99 , '99');
> {code}
> exit beeline and connect with user another 
> {code}
> ./beeline -u 
> "jdbc:hive2://localhost:1/default?hive.strict.checks.cartesian.product=false;hive.txn.timeout=4s;hive.txn.heartbeat.threadpool.size=1;hadoop.security.credential.provider.path=jceks://hdfs/user/another/another.jceks;ssl.server.keystore.keypassword=another"
>  -n another
> create table another_a_acid (i int, b string) clustered by (i) into 8 buckets 
> stored as orc tblproperties('transactional'='true');
> insert overwrite table another_a_acid select a2.i, a3.b from a a1 join a a2 
> join a a3 on 1=1;
> {code}
> open another beeline session with user test:
> {code}
> ./beeline -u 
> "jdbc:hive2://localhost:1/default?hive.strict.checks.cartesian.product=false;hive.txn.timeout=4s;hive.txn.heartbeat.threadpool.size=1;hadoop.security.credential.provider.path=jceks://hdfs/user/test/test.jceks;ssl.server.keystore.keypassword=test"
>  -n test
> create table a_acid (i int, b string) clustered by (i) into 8 buckets stored 
> as orc tblproperties('transactional'='true');
> insert overwrite table a_acid select a2.i, a3.b from a a1 join a a2 join a a3 
> on 1=1;
> {code}
> fails with exception 
> {code}
> 2017-11-17T12:15:52,664 DEBUG [Heartbeater-1] retry.RetryInvocationHandler: 
> Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over 
> null. Not retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException: Permission denied: user=test, 
> access=EXECUTE, inode="/user/another/another.jceks":another:another:drwx--
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
>   at 
> 

[jira] [Commented] (HIVE-18090) acid heartbeat fails when metastore is connected via hadoop credential

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261997#comment-16261997
 ] 

anishek commented on HIVE-18090:


Thanks for the review [~ekoifman], Fixed the typos in "Description", Going to 
do a quick look at tests failures once before i commit, cant access the apache 
logs.

> acid heartbeat fails when metastore is connected via hadoop credential
> --
>
> Key: HIVE-18090
> URL: https://issues.apache.org/jira/browse/HIVE-18090
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Transactions
>Affects Versions: 1.3.0, 2.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-18090.0.patch
>
>
> steps to recreate the issue. assuming two users 
> * test
> * another 
> create two jceks files for each user and place them on hdfs with access to 
> that file only allowed to the user. hdfs locations with permissions 
> {code}
> -rwx--   1 another another492 2017-11-16 13:06 
> /user/another/another.jceks
> -rwx--   1 test test489 2017-11-16 13:05 /user/test/test.jceks
> {code}
> password used to create 
> * /user/another/another.jceks -- another
> * /user/test/test.jceks -- test
> on core-site.xml 
> {code}
> 
> hadoop.proxyuser.[superuser].hosts
> *
> 
> 
> hadoop.proxyuser.[superuser].groups
> *
> 
> {code}
> and restart hdfs.
> enable ACID on HS2 (change the required properties).additional changes on  
> hiveserver2 configs 
> {code}
> * hive.metastore.warehouse.dir=file:///tmp/hive/test-warehouse
> * hive.server2.enable.doAs=true
> * remove javax.jdo.option.ConnectionPassword property from hive-site.xml
> {code}
> start hiveserver2
> connect to the server using beeline using any user:
> {code}
> create table a (i int, b string);
> insert into a values (0 , '0'), (1 , '1'), (2 , '2'), (3 , '3'), (4 , '4'), 
> (5 , '5'), (6 , '6'), (7 , '7'), (8 , '8'), (9 , '9'), (10 , '10'), (11 , 
> '11'), (12 , '12'), (13 , '13'), (14 , '14'), (15 , '15'), (16 , '16'), (17 , 
> '17'), (18 , '18'), (19 , '19'), (20 , '20'), (21 , '21'), (22 , '22'), (23 , 
> '23'), (24 , '24'), (25 , '25'), (26 , '26'), (27 , '27'), (28 , '28'), (29 , 
> '29'), (30 , '30'), (31 , '31'), (32 , '32'), (33 , '33'), (34 , '34'), (35 , 
> '35'), (36 , '36'), (37 , '37'), (38 , '38'), (39 , '39'), (40 , '40'), (41 , 
> '41'), (42 , '42'), (43 , '43'), (44 , '44'), (45 , '45'), (46 , '46'), (47 , 
> '47'), (48 , '48'), (49 , '49'), (50 , '50'), (51 , '51'), (52 , '52'), (53 , 
> '53'), (54 , '54'), (55 , '55'), (56 , '56'), (57 , '57'), (58 , '58'), (59 , 
> '59'), (60 , '60'), (61 , '61'), (62 , '62'), (63 , '63'), (64 , '64'), (65 , 
> '65'), (66 , '66'), (67 , '67'), (68 , '68'), (69 , '69'), (70 , '70'), (71 , 
> '71'), (72 , '72'), (73 , '73'), (74 , '74'), (75 , '75'), (76 , '76'), (77 , 
> '77'), (78 , '78'), (79 , '79'), (80 , '80'), (81 , '81'), (82 , '82'), (83 , 
> '83'), (84 , '84'), (85 , '85'), (86 , '86'), (87 , '87'), (88 , '88'), (89 , 
> '89'), (90 , '90'), (91 , '91'), (92 , '92'), (93 , '93'), (94 , '94'), (95 , 
> '95'), (96 , '96'), (97 , '97'), (98 , '98'), (99 , '99');
> {code}
> exit beeline and connect with user another 
> {code}
> ./beeline -u 
> "jdbc:hive2://localhost:1/default?hive.strict.checks.cartesian.product=false;hive.txn.timeout=4s;hive.txn.heartbeat.threadpool.size=1;hadoop.security.credential.provider.path=jceks://hdfs/user/another/another.jceks;ssl.server.keystore.keypassword=another"
>  -n another
> create table another_a_acid (i int, b string) clustered by (i) into 8 buckets 
> stored as orc tblproperties('transactional'='true');
> insert overwrite table another_a_acid select a2.i, a3.b from a a1 join a a2 
> join a a3 on 1=1;
> {code}
> open another beeline session with user test:
> {code}
> ./beeline -u 
> "jdbc:hive2://localhost:1/default?hive.strict.checks.cartesian.product=false;hive.txn.timeout=4s;hive.txn.heartbeat.threadpool.size=1;hadoop.security.credential.provider.path=jceks://hdfs/user/test/test.jceks;ssl.server.keystore.keypassword=test"
>  -n test
> create table a_acid (i int, b string) clustered by (i) into 8 buckets stored 
> as orc tblproperties('transactional'='true');
> insert overwrite table a_acid select a2.i, a3.b from a a1 join a a2 join a a3 
> on 1=1;
> {code}
> fails with exception 
> {code}
> 2017-11-17T12:15:52,664 DEBUG [Heartbeater-1] retry.RetryInvocationHandler: 
> Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over 
> null. Not retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException: Permission denied: user=test, 
> access=EXECUTE, inode="/user/another/another.jceks":another:another:drwx--
>   at 
> 

[jira] [Updated] (HIVE-18090) acid heartbeat fails when metastore is connected via hadoop credential

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-18090:
---
Description: 
steps to recreate the issue. assuming two users 
* test
* another 

create two jceks files for each user and place them on hdfs with access to that 
file only allowed to the user. hdfs locations with permissions 
{code}
-rwx--   1 another another492 2017-11-16 13:06 
/user/another/another.jceks
-rwx--   1 test test489 2017-11-16 13:05 /user/test/test.jceks
{code}

password used to create 
* /user/another/another.jceks -- another
* /user/test/test.jceks -- test

on core-site.xml 
{code}

hadoop.proxyuser.[superuser].hosts
*


hadoop.proxyuser.[superuser].groups
*

{code}
and restart hdfs.
enable ACID on HS2 (change the required properties).additional changes on  
hiveserver2 configs 
{code}
* hive.metastore.warehouse.dir=file:///tmp/hive/test-warehouse
* hive.server2.enable.doAs=true
* remove javax.jdo.option.ConnectionPassword property from hive-site.xml
{code}
start hiveserver2

connect to the server using beeline using any user:
{code}
create table a (i int, b string);
insert into a values (0 , '0'), (1 , '1'), (2 , '2'), (3 , '3'), (4 , '4'), (5 
, '5'), (6 , '6'), (7 , '7'), (8 , '8'), (9 , '9'), (10 , '10'), (11 , '11'), 
(12 , '12'), (13 , '13'), (14 , '14'), (15 , '15'), (16 , '16'), (17 , '17'), 
(18 , '18'), (19 , '19'), (20 , '20'), (21 , '21'), (22 , '22'), (23 , '23'), 
(24 , '24'), (25 , '25'), (26 , '26'), (27 , '27'), (28 , '28'), (29 , '29'), 
(30 , '30'), (31 , '31'), (32 , '32'), (33 , '33'), (34 , '34'), (35 , '35'), 
(36 , '36'), (37 , '37'), (38 , '38'), (39 , '39'), (40 , '40'), (41 , '41'), 
(42 , '42'), (43 , '43'), (44 , '44'), (45 , '45'), (46 , '46'), (47 , '47'), 
(48 , '48'), (49 , '49'), (50 , '50'), (51 , '51'), (52 , '52'), (53 , '53'), 
(54 , '54'), (55 , '55'), (56 , '56'), (57 , '57'), (58 , '58'), (59 , '59'), 
(60 , '60'), (61 , '61'), (62 , '62'), (63 , '63'), (64 , '64'), (65 , '65'), 
(66 , '66'), (67 , '67'), (68 , '68'), (69 , '69'), (70 , '70'), (71 , '71'), 
(72 , '72'), (73 , '73'), (74 , '74'), (75 , '75'), (76 , '76'), (77 , '77'), 
(78 , '78'), (79 , '79'), (80 , '80'), (81 , '81'), (82 , '82'), (83 , '83'), 
(84 , '84'), (85 , '85'), (86 , '86'), (87 , '87'), (88 , '88'), (89 , '89'), 
(90 , '90'), (91 , '91'), (92 , '92'), (93 , '93'), (94 , '94'), (95 , '95'), 
(96 , '96'), (97 , '97'), (98 , '98'), (99 , '99');
{code}

exit beeline and connect with user another 
{code}
./beeline -u 
"jdbc:hive2://localhost:1/default?hive.strict.checks.cartesian.product=false;hive.txn.timeout=4s;hive.txn.heartbeat.threadpool.size=1;hadoop.security.credential.provider.path=jceks://hdfs/user/another/another.jceks;ssl.server.keystore.keypassword=another"
 -n another

create table another_a_acid (i int, b string) clustered by (i) into 8 buckets 
stored as orc tblproperties('transactional'='true');

insert overwrite table another_a_acid select a2.i, a3.b from a a1 join a a2 
join a a3 on 1=1;
{code}

open another beeline session with user test:
{code}
./beeline -u 
"jdbc:hive2://localhost:1/default?hive.strict.checks.cartesian.product=false;hive.txn.timeout=4s;hive.txn.heartbeat.threadpool.size=1;hadoop.security.credential.provider.path=jceks://hdfs/user/test/test.jceks;ssl.server.keystore.keypassword=test"
 -n test

create table a_acid (i int, b string) clustered by (i) into 8 buckets stored as 
orc tblproperties('transactional'='true');

insert overwrite table a_acid select a2.i, a3.b from a a1 join a a2 join a a3 
on 1=1;
{code}

fails with exception 
{code}
2017-11-17T12:15:52,664 DEBUG [Heartbeater-1] retry.RetryInvocationHandler: 
Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over 
null. Not retrying because try once and fail.
org.apache.hadoop.ipc.RemoteException: Permission denied: user=test, 
access=EXECUTE, inode="/user/another/another.jceks":another:another:drwx--
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1955)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:109)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4111)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1137)
at 

[jira] [Updated] (HIVE-18043) Vectorization: Support List type in MapWork

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated HIVE-18043:

Attachment: HIVE-18043.002.patch

Update the patch to fix the vector_complex_join.q, which means to support 
complex join for vectorization.

> Vectorization: Support List type in MapWork
> ---
>
> Key: HIVE-18043
> URL: https://issues.apache.org/jira/browse/HIVE-18043
> Project: Hive
>  Issue Type: Improvement
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18043.001.patch, HIVE-18043.002.patch
>
>
> Support Complex Types in vectorization is finished in HIVE-16589, but List 
> type is still not support in MapWork. It should be supported to improve the 
> performance when vectorization is enable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-335) External Tables should have the option to be marked Read Only

[Gaurav Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261985#comment-16261985
 ] 

Gaurav Shah commented on HIVE-335:
--

is there a workaround for this ? trying to create readonly external table since 
I do not want end users to be able to impact prod data

> External Tables should have the option to be marked Read Only
> -
>
> Key: HIVE-335
> URL: https://issues.apache.org/jira/browse/HIVE-335
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore, Query Processor
>Reporter: Richard Lee
>
> When creating an External Table, it'd be awesome to have the option of NOT 
> allowing writes to it (disallow any INSERTs or if hive ever allows UPDATEs).  
> Adding and Dropping Partitions should still be allowed.
> This will enable hive to "play well" with external data stores other than 
> hdfs where data should be non-maleable.
> I'd recomend the following syntax, which applies ONLY to external tables:
> CREATE EXTERNAL [READONLY] TABLE ...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18111) Fix temp path for Spark DPP sink

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261984#comment-16261984
 ] 

Hive QA commented on HIVE-18111:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898780/HIVE-18111.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11410 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7953/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7953/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7953/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898780 - PreCommit-HIVE-Build

> Fix temp path for Spark DPP sink
> 
>
> Key: HIVE-18111
> URL: https://issues.apache.org/jira/browse/HIVE-18111
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-18111.1.patch, HIVE-18111.2.patch, 
> HIVE-18111.3.patch
>
>
> Before HIVE-17877, each DPP sink has only one target work. The output path of 
> a DPP work is {{TMP_PATH/targetWorkId/dppWorkId}}. When we do the pruning, 
> each map work reads DPP outputs under {{TMP_PATH/targetWorkId}}.
> After HIVE-17877, each DPP sink can have multiple target works. It's possible 
> that a map work needs to read DPP outputs from multiple 
> {{TMP_PATH/targetWorkId}}. To solve this, I think we can have a DPP output 
> path specific to each query, e.g. {{QUERY_TMP_PATH/dpp_output}}. Each DPP 
> work outputs to {{QUERY_TMP_PATH/dpp_output/dppWorkId}}. And each map work 
> reads from {{QUERY_TMP_PATH/dpp_output}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18123) Explain formatted improve column expression map display

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18123:
---
Status: Patch Available  (was: Open)

> Explain formatted improve column expression map display
> ---
>
> Key: HIVE-18123
> URL: https://issues.apache.org/jira/browse/HIVE-18123
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18123.1.patch, HIVE-18123.2.patch
>
>
> HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that 
> map was a little off. This jira is to improve the formatting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18123) Explain formatted improve column expression map display

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18123:
---
Attachment: HIVE-18123.2.patch

> Explain formatted improve column expression map display
> ---
>
> Key: HIVE-18123
> URL: https://issues.apache.org/jira/browse/HIVE-18123
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18123.1.patch, HIVE-18123.2.patch
>
>
> HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that 
> map was a little off. This jira is to improve the formatting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18123) Explain formatted improve column expression map display

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18123:
---
Status: Open  (was: Patch Available)

> Explain formatted improve column expression map display
> ---
>
> Key: HIVE-18123
> URL: https://issues.apache.org/jira/browse/HIVE-18123
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18123.1.patch
>
>
> HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that 
> map was a little off. This jira is to improve the formatting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18126) Mechanics of multiple commands with OVERWRITE in a singe transactions

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261971#comment-16261971
 ] 

Sergey Shelukhin commented on HIVE-18126:
-

Also what about insert overwrite with union in tez?

> Mechanics of multiple commands with OVERWRITE in a singe transactions
> -
>
> Key: HIVE-18126
> URL: https://issues.apache.org/jira/browse/HIVE-18126
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> for Insert overwrite/load data overwrite we create base_x/ to hold the data 
> thus are able to make Overwrite command non-blocking.  
> What happens if multiple IOWs are run against the same table/partition in the 
> same transaction.
> should base support a suffix base_x_000 like deltas?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18109) fix identifier usage in parser

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18109:

Status: Patch Available  (was: Open)

> fix identifier usage in parser
> --
>
> Key: HIVE-18109
> URL: https://issues.apache.org/jira/browse/HIVE-18109
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18109.patch
>
>
> HIVE-17902 -broke- exposed this ;)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18073) AM may assert when duck count for it is reduced

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18073:

Status: Patch Available  (was: Open)

> AM may assert when duck count for it is reduced
> ---
>
> Key: HIVE-18073
> URL: https://issues.apache.org/jira/browse/HIVE-18073
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18073.patch
>
>
> Sometimes it asserts that it doesn't have so many ducks to give away. This 
> should never happen, need to debug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18109) fix identifier usage in parser

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18109:

Description: HIVE-17902 -broke- exposed this ;)  (was: HIVE-17902 broke 
this)

> fix identifier usage in parser
> --
>
> Key: HIVE-18109
> URL: https://issues.apache.org/jira/browse/HIVE-18109
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18109.patch
>
>
> HIVE-17902 -broke- exposed this ;)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18109) Don't reserve pool and default as keywords

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18109:

Attachment: HIVE-18109.patch

Fixed here and in some other places. Didn't actually run the tests cause I'm 
running something else :) 
[~ashutoshc] [~ekoifman] does this make sense?

> Don't reserve pool and default as keywords
> --
>
> Key: HIVE-18109
> URL: https://issues.apache.org/jira/browse/HIVE-18109
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18109.patch
>
>
> HIVE-17902 broke this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18109) fix identifier usage in parser

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18109:

Summary: fix identifier usage in parser  (was: Don't reserve pool and 
default as keywords)

> fix identifier usage in parser
> --
>
> Key: HIVE-18109
> URL: https://issues.apache.org/jira/browse/HIVE-18109
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18109.patch
>
>
> HIVE-17902 broke this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18073) AM may assert when duck count for it is reduced

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18073:

Attachment: HIVE-18073.patch

The duck is used for the same task multiple times when scheduling is not 
possible, resulting in the reduction of the available count over time.

> AM may assert when duck count for it is reduced
> ---
>
> Key: HIVE-18073
> URL: https://issues.apache.org/jira/browse/HIVE-18073
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18073.patch
>
>
> Sometimes it asserts that it doesn't have so many ducks to give away. This 
> should never happen, need to debug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18109) Don't reserve pool and default as keywords

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261956#comment-16261956
 ] 

Sergey Shelukhin commented on HIVE-18109:
-

The definition of show locks is:
{noformat}
| KW_SHOW KW_LOCKS 
  (
  (KW_DATABASE|KW_SCHEMA) => (KW_DATABASE|KW_SCHEMA) (dbName=Identifier) 
(isExtended=KW_EXTENDED)? -> ^(TOK_SHOWDBLOCKS $dbName $isExtended?)
  |
  (parttype=partTypeExpr)? (isExtended=KW_EXTENDED)? -> ^(TOK_SHOWLOCKS 
$parttype? $isExtended?)
  )
{noformat}

I think the problem is dbName=Identifier here; because
{noformat}
identifier
:
Identifier
| nonReserved -> Identifier[$nonReserved.start]
;
{noformat}
but Identifier is just some letters and numbers and stuff.


> Don't reserve pool and default as keywords
> --
>
> Key: HIVE-18109
> URL: https://issues.apache.org/jira/browse/HIVE-18109
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
>
> HIVE-17902 broke this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261953#comment-16261953
 ] 

Hive QA commented on HIVE-18127:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898769/HIVE-18127.1.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11413 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[add_jar_with_file_removed]
 (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testThatCliDriverDoesNotStripComments
 (batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=102)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39] 
(batchId=246)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7952/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7952/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7952/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898769 - PreCommit-HIVE-Build

> Do not strip '--' comments from shell commands issued from CliDriver
> 
>
> Key: HIVE-18127
> URL: https://issues.apache.org/jira/browse/HIVE-18127
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18127.1.patch
>
>
> CLiDriver has the ability to run shell commands by prefixing them with '!".
> This behavior is not widely used (there are only 3 examples in .q files).
> Since HIVE-16935 started stripping comments starting with '\-\-', a shell 
> command containing '--' will not work correctly.
> Fix this by using the unstripped command for shell commands.
> Note that it would be a security hole for HS2 to allow execution of arbitrary 
> shell commands from a client command.
> Add tests to nail down correct behavior with '--' comments:
> * CliDriver should not strip strings starting with '--' in a shell command 
> (FIXED in this change).
> * HiveCli should strip '--' comments.
> * A Jdbc program should allow commands starting with "!" but these will fail 
> in the sql parser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18100) Some tests time out

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-18100:

Status: Patch Available  (was: Open)

> Some tests time out
> ---
>
> Key: HIVE-18100
> URL: https://issues.apache.org/jira/browse/HIVE-18100
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-18100.1.patch, HIVE-18100.2.patch, 
> HIVE-18100.3.patch, HIVE-18100.patch
>
>
> Some tests had 100s of queries in a single query which times out resulting in 
> Hive QA failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18100) Some tests time out

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-18100:

Attachment: HIVE-18100.3.patch

> Some tests time out
> ---
>
> Key: HIVE-18100
> URL: https://issues.apache.org/jira/browse/HIVE-18100
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-18100.1.patch, HIVE-18100.2.patch, 
> HIVE-18100.3.patch, HIVE-18100.patch
>
>
> Some tests had 100s of queries in a single query which times out resulting in 
> Hive QA failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18100) Some tests time out

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-18100:

Status: Open  (was: Patch Available)

> Some tests time out
> ---
>
> Key: HIVE-18100
> URL: https://issues.apache.org/jira/browse/HIVE-18100
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-18100.1.patch, HIVE-18100.2.patch, HIVE-18100.patch
>
>
> Some tests had 100s of queries in a single query which times out resulting in 
> Hive QA failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18109) Don't reserve pool and default as keywords

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261923#comment-16261923
 ] 

Eugene Koifman commented on HIVE-18109:
---

This is the error for dbtxnmgr_showlocks
{noformat}
show locks database default
2017-11-21T19:26:04,553 ERROR [d31f1352-252c-4248-9859-462012e4a065 main] 
ql.Driver: FAILED: ParseException line 3:20 mismatched input 'default' 
expecting Identifier\
 near 'database' in show statement
org.apache.hadoop.hive.ql.parse.ParseException: line 3:20 mismatched input 
'default' expecting Identifier near 'database' in show statement
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:211)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:67)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:544)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1457)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1589)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1356)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1346)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:342)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1286)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1260)
at 
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:173)
at 
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
{noformat}

> Don't reserve pool and default as keywords
> --
>
> Key: HIVE-18109
> URL: https://issues.apache.org/jira/browse/HIVE-18109
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
>
> HIVE-17902 broke this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17361:
--
Attachment: HIVE-17361.20.patch

> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, 
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, 
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, 
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, 
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.3.patch, 
> HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap 
> between ACID table and regular hive table.
> Current Documentation is under [DML 
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
>  and [Loading files into 
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it 
> uses the input file name which may not be in 0_0 which can break some 
> read logic.  (Certainly will for Acid).
> * It does not check the schema of the file.  This may be a non issue for Acid 
> which requires ORC which is self describing so Schema Evolution may handle 
> this seamlessly.  (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible. 
> * Bucketed (and thus sorted) tables don't support Load Data (but only if 
> hive.strict.checks.bucketing=true (default)).  Will keep this restriction for 
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a 
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created before files are moved, delta_8_8 
> becomes visible before it's populated.  Is that an issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18123) Explain formatted improve column expression map display

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261910#comment-16261910
 ] 

Hive QA commented on HIVE-18123:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898764/HIVE-18123.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11410 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_1_23] 
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input4] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join0] (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join4] 
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6] 
(batchId=41)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=102)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7951/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7951/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7951/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898764 - PreCommit-HIVE-Build

> Explain formatted improve column expression map display
> ---
>
> Key: HIVE-18123
> URL: https://issues.apache.org/jira/browse/HIVE-18123
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18123.1.patch
>
>
> HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that 
> map was a little off. This jira is to improve the formatting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18111) Fix temp path for Spark DPP sink

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-18111:
--
Attachment: HIVE-18111.3.patch

> Fix temp path for Spark DPP sink
> 
>
> Key: HIVE-18111
> URL: https://issues.apache.org/jira/browse/HIVE-18111
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-18111.1.patch, HIVE-18111.2.patch, 
> HIVE-18111.3.patch
>
>
> Before HIVE-17877, each DPP sink has only one target work. The output path of 
> a DPP work is {{TMP_PATH/targetWorkId/dppWorkId}}. When we do the pruning, 
> each map work reads DPP outputs under {{TMP_PATH/targetWorkId}}.
> After HIVE-17877, each DPP sink can have multiple target works. It's possible 
> that a map work needs to read DPP outputs from multiple 
> {{TMP_PATH/targetWorkId}}. To solve this, I think we can have a DPP output 
> path specific to each query, e.g. {{QUERY_TMP_PATH/dpp_output}}. Each DPP 
> work outputs to {{QUERY_TMP_PATH/dpp_output/dppWorkId}}. And each map work 
> reads from {{QUERY_TMP_PATH/dpp_output}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261859#comment-16261859
 ] 

Hive QA commented on HIVE-17361:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898756/HIVE-17361.19.patch

{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 11418 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_default] (batchId=81)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=102)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7950/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7950/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7950/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898756 - PreCommit-HIVE-Build

> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, 
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, 
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, 
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, 
> HIVE-17361.2.patch, HIVE-17361.3.patch, HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap 
> between ACID table and regular hive table.
> Current Documentation is under [DML 
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
>  and [Loading files into 
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it 
> uses the input file name which may not be in 0_0 which can break some 
> read logic.  (Certainly will for Acid).
> * It does not check the schema of the file.  This may be a non issue for Acid 
> which requires ORC which is self describing so Schema Evolution may handle 
> this seamlessly.  (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible. 
> * Bucketed (and thus sorted) tables don't support Load Data (but only if 
> hive.strict.checks.bucketing=true (default)).  Will keep this restriction for 
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a 
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created before files are moved, delta_8_8 
> becomes visible before it's populated.  Is that an issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18054) Make Lineage work with concurrent queries on a Session

[Andrew Sherman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261838#comment-16261838
 ] 

Andrew Sherman commented on HIVE-18054:
---

To fix this, move LineageState from SessionState to QueryState.
In MoveTask/MoveWork use the LineageState from the MoveTask's QueryState
rather than trying to use it from MoveWork.
Add a test which runs multiple jdbc queries in a thread pool
against the same connection and show that Vertices are not lost from Lineage.
As part of this test, add ReadableHook, an ExecuteWithHookContext that stores
HookContexts in memory and makes them available for reading.
Make LineageLogger methods static so they can be used elsewhere.

Sometimes a running query (originating in a Driver) will instantiate
another Driver to run or compile another query. Because these Drivers
shared a Session, the child Driver would accumulate Lineage information
along with that of the parent Driver. For consistency a LineageState is
passed to these child Drivers and stored in the new Driver's QueryState.


>  Make Lineage work with concurrent queries on a Session
> ---
>
> Key: HIVE-18054
> URL: https://issues.apache.org/jira/browse/HIVE-18054
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18054.1.patch, HIVE-18054.2.patch, 
> HIVE-18054.3.patch, HIVE-18054.4.patch, HIVE-18054.5.patch, 
> HIVE-18054.6.patch, HIVE-18054.7.patch
>
>
> A Hive Session can contain multiple concurrent sql Operations.
> Lineage is currently tracked in SessionState and is cleared when a query 
> completes. This results in Lineage for other running queries being lost.
> To fix this, move LineageState from SessionState to QueryState.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18054) Make Lineage work with concurrent queries on a Session

[Andrew Sherman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261837#comment-16261837
 ] 

Andrew Sherman commented on HIVE-18054:
---

Test failures look unrelated.
[~stakiar] can you take a look at this change? Thanks

>  Make Lineage work with concurrent queries on a Session
> ---
>
> Key: HIVE-18054
> URL: https://issues.apache.org/jira/browse/HIVE-18054
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18054.1.patch, HIVE-18054.2.patch, 
> HIVE-18054.3.patch, HIVE-18054.4.patch, HIVE-18054.5.patch, 
> HIVE-18054.6.patch, HIVE-18054.7.patch
>
>
> A Hive Session can contain multiple concurrent sql Operations.
> Lineage is currently tracked in SessionState and is cleared when a query 
> completes. This results in Lineage for other running queries being lost.
> To fix this, move LineageState from SessionState to QueryState.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-18127:
--
Attachment: HIVE-18127.1.patch

> Do not strip '--' comments from shell commands issued from CliDriver
> 
>
> Key: HIVE-18127
> URL: https://issues.apache.org/jira/browse/HIVE-18127
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18127.1.patch
>
>
> CLiDriver has the ability to run shell commands by prefixing them with '!".
> This behavior is not widely used (there are only 3 examples in .q files).
> Since HIVE-16935 started stripping comments starting with '\-\-', a shell 
> command containing '--' will not work correctly.
> Fix this by using the unstripped command for shell commands.
> Note that it would be a security hole for HS2 to allow execution of arbitrary 
> shell commands from a client command.
> Add tests to nail down correct behavior with '--' comments:
> * CliDriver should not strip strings starting with '--' in a shell command 
> (FIXED in this change).
> * HiveCli should strip '--' comments.
> * A Jdbc program should allow commands starting with "!" but these will fail 
> in the sql parser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-18127:
--
Status: Patch Available  (was: Open)

> Do not strip '--' comments from shell commands issued from CliDriver
> 
>
> Key: HIVE-18127
> URL: https://issues.apache.org/jira/browse/HIVE-18127
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18127.1.patch
>
>
> CLiDriver has the ability to run shell commands by prefixing them with '!".
> This behavior is not widely used (there are only 3 examples in .q files).
> Since HIVE-16935 started stripping comments starting with '\-\-', a shell 
> command containing '--' will not work correctly.
> Fix this by using the unstripped command for shell commands.
> Note that it would be a security hole for HS2 to allow execution of arbitrary 
> shell commands from a client command.
> Add tests to nail down correct behavior with '--' comments:
> * CliDriver should not strip strings starting with '--' in a shell command 
> (FIXED in this change).
> * HiveCli should strip '--' comments.
> * A Jdbc program should allow commands starting with "!" but these will fail 
> in the sql parser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18128) Setting AcidUtils.setTransactionalTableScan in HiveInputFormat causes downstream errors

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18128:
--
Attachment: HIVE-18128.WIP.patch

> Setting AcidUtils.setTransactionalTableScan in HiveInputFormat causes 
> downstream errors
> ---
>
> Key: HIVE-18128
> URL: https://issues.apache.org/jira/browse/HIVE-18128
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
> Attachments: HIVE-18128.WIP.patch
>
>
> This should really be set in addSplitsForGroup().  See attached patch for 
> details



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman reassigned HIVE-18127:
-


> Do not strip '--' comments from shell commands issued from CliDriver
> 
>
> Key: HIVE-18127
> URL: https://issues.apache.org/jira/browse/HIVE-18127
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>
> CLiDriver has the ability to run shell commands by prefixing them with '!".
> This behavior is not widely used (there are only 3 examples in .q files).
> Since HIVE-16935 started stripping comments starting with '\-\-', a shell 
> command containing '--' will not work correctly.
> Fix this by using the unstripped command for shell commands.
> Note that it would be a security hole for HS2 to allow execution of arbitrary 
> shell commands from a client command.
> Add tests to nail down correct behavior with '--' comments:
> * CliDriver should not strip strings starting with '--' in a shell command 
> (FIXED in this change).
> * HiveCli should strip '--' comments.
> * A Jdbc program should allow commands starting with "!" but these will fail 
> in the sql parser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18126) Mechanics of multiple commands with OVERWRITE in a singe transactions

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-18126:
-


> Mechanics of multiple commands with OVERWRITE in a singe transactions
> -
>
> Key: HIVE-18126
> URL: https://issues.apache.org/jira/browse/HIVE-18126
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> for Insert overwrite/load data overwrite we create base_x/ to hold the data 
> thus are able to make Overwrite command non-blocking.  
> What happens if multiple IOWs are run against the same table/partition in the 
> same transaction.
> should base support a suffix base_x_000 like deltas?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18125) Support arbitrary file names in input to Load Data

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-18125:
-


> Support arbitrary file names in input to Load Data
> --
>
> Key: HIVE-18125
> URL: https://issues.apache.org/jira/browse/HIVE-18125
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> HIVE-17361 only allows 0_0 and _0_copy_1.  Should it support 
> arbitrary names?
> If so, should it sort them and rename _0, 0001_0, etc?
> This is probably a lot easier than changing the whole code base to assign 
> proper 'bucket' (writerId) everywhere Acid reads such file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261803#comment-16261803
 ] 

Hive QA commented on HIVE-17980:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898754/HIVE-17980.patch

{color:green}SUCCESS:{color} +1 due to 28 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11354 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_product_check_2] 
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=223)
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
 (batchId=225)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
 (batchId=225)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
 (batchId=236)
org.apache.hive.hcatalog.api.TestHCatClient.org.apache.hive.hcatalog.api.TestHCatClient
 (batchId=185)
org.apache.hive.hcatalog.api.TestHCatClientNotification.org.apache.hive.hcatalog.api.TestHCatClientNotification
 (batchId=237)
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
 (batchId=185)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.org.apache.hive.hcatalog.listener.TestDbNotificationListener
 (batchId=237)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=229)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=229)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=229)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7949/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7949/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7949/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898754 - PreCommit-HIVE-Build

> Move HiveMetaStoreClient plus a few remaining classes.
> --
>
> Key: HIVE-17980
> URL: https://issues.apache.org/jira/browse/HIVE-17980
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17980.patch
>
>
> This moves over HiveMetaStoreClient a few more remaining things that haven't 
> been moved yet.  It also cleans up the mess I made of MetaStoreUtils.  I 
> renamed the class in metastore to HiveMetaStoreUtils and left a few methods 
> in there that could be moved.  Everything else has been moved to 
> standalone-metastore/.../MetaStoreUtils.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17232) "No match found" Compactor finds a bucket file thinking it's a directory

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-17232:
-

Assignee: Eugene Koifman  (was: Steve Yeom)

>  "No match found"  Compactor finds a bucket file thinking it's a directory
> --
>
> Key: HIVE-17232
> URL: https://issues.apache.org/jira/browse/HIVE-17232
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-17232.01.patch
>
>
> {noformat}
> 2017-08-02T12:38:11,996  WARN [main] compactor.CompactorMR: Found a 
> non-bucket file that we thought matched the bucket pattern! 
> file:/Users/ekoifman/dev/hiv\
> erwgit/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands2-1501702264311/warehouse/acidtblpart/p=1/delta_013_013_/bucket_1
>  Matcher=java\
> .util.regex.Matcher[pattern=^[0-9]{6} region=0,12 lastmatch=]
> 2017-08-02T12:38:11,996  INFO [main] mapreduce.JobSubmitter: Cleaning up the 
> staging area 
> file:/tmp/hadoop/mapred/staging/ekoifman1723152463/.staging/job_lo\
> cal1723152463_0183
> 2017-08-02T12:38:11,997 ERROR [main] compactor.Worker: Caught exception while 
> trying to compact 
> id:1,dbname:default,tableName:ACIDTBLPART,partName:null,stat\
> e:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0.
>   Marking failed to avoid repeated failures, java.lang.IllegalStateException: 
> \
> No match found
> at java.util.regex.Matcher.group(Matcher.java:536)
> at java.util.regex.Matcher.group(Matcher.java:496)
> at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorInputFormat.addFileToMap(CompactorMR.java:577)
> at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorInputFormat.getSplits(CompactorMR.java:549)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:330)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:322)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:198)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at 
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
> at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.launchCompactionJob(CompactorMR.java:320)
> at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:275)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:166)
> at 
> org.apache.hadoop.hive.ql.TestTxnCommands2.runWorker(TestTxnCommands2.java:1138)
> at 
> org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned(TestTxnCommands2.java:894)
> {noformat}
> the stack trace points to 1st runWorker() in updateDeletePartitioned() though 
> the test run was TestTxnCommands2WithSplitUpdateAndVectorization



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18123) Explain formatted improve column expression map display

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18123:
---
Attachment: HIVE-18123.1.patch

> Explain formatted improve column expression map display
> ---
>
> Key: HIVE-18123
> URL: https://issues.apache.org/jira/browse/HIVE-18123
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18123.1.patch
>
>
> HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that 
> map was a little off. This jira is to improve the formatting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18123) Explain formatted improve column expression map display

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-18123:
---
Status: Patch Available  (was: Open)

> Explain formatted improve column expression map display
> ---
>
> Key: HIVE-18123
> URL: https://issues.apache.org/jira/browse/HIVE-18123
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18123.1.patch
>
>
> HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that 
> map was a little off. This jira is to improve the formatting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18124) clean up isAcidTable() API

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-18124:
-


>  clean up isAcidTable() API
> ---
>
> Key: HIVE-18124
> URL: https://issues.apache.org/jira/browse/HIVE-18124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> With the addition of MM tables (_AcidUtils.isInsertOnlyTable(table)_) the 
> methods in AcidUtils and dependent places are very muddled.
> Need to clean it up so that there is a isTransactional(Table) that checks 
> transactional=true setting and isAcid(Table) to mean full ACID and 
> isInsertOnly(Table) to mean MM tables.
> This would accurately describe the semantics of the tables.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18123) Explain formatted improve column expression map display

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg reassigned HIVE-18123:
--


> Explain formatted improve column expression map display
> ---
>
> Key: HIVE-18123
> URL: https://issues.apache.org/jira/browse/HIVE-18123
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>
> HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that 
> map was a little off. This jira is to improve the formatting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17361:
--
Attachment: HIVE-17361.19.patch

> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, 
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, 
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, 
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, 
> HIVE-17361.2.patch, HIVE-17361.3.patch, HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap 
> between ACID table and regular hive table.
> Current Documentation is under [DML 
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
>  and [Loading files into 
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it 
> uses the input file name which may not be in 0_0 which can break some 
> read logic.  (Certainly will for Acid).
> * It does not check the schema of the file.  This may be a non issue for Acid 
> which requires ORC which is self describing so Schema Evolution may handle 
> this seamlessly.  (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible. 
> * Bucketed (and thus sorted) tables don't support Load Data (but only if 
> hive.strict.checks.bucketing=true (default)).  Will keep this restriction for 
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a 
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created before files are moved, delta_8_8 
> becomes visible before it's populated.  Is that an issue?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261717#comment-16261717
 ] 

Hive QA commented on HIVE-18076:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898751/HIVE-18076.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 11397 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_ssb]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=102)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1 
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
 (batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
 (batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
 (batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles 
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent 
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead 
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes 
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
 (batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
 (batchId=231)
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
 (batchId=245)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7948/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7948/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7948/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898751 - PreCommit-HIVE-Build

> killquery doesn't actually work for non-trigger WM kills, or the error 
> message is not propagated
> 
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice 
> message like triggers do.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-11294) Use HBase to cache aggregated stats

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-11294:
--
Labels: pull-request-available  (was: )

> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: hbase-metastore-branch, 2.0.0
>
> Attachments: HIVE-11294.2.patch, HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261691#comment-16261691
 ] 

ASF GitHub Bot commented on HIVE-11294:
---

Github user alanfgates closed the pull request at:

https://github.com/apache/hive/pull/46


> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: hbase-metastore-branch, 2.0.0
>
> Attachments: HIVE-11294.2.patch, HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client.  Given that 
> HBase can easily manage the scale of caching aggregated stats we should be 
> using it to do so.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17234) Remove HBase metastore from master

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-17234:
--
Labels: pull-request-available  (was: )

> Remove HBase metastore from master
> --
>
> Key: HIVE-17234
> URL: https://issues.apache.org/jira/browse/HIVE-17234
> Project: Hive
>  Issue Type: Task
>  Components: HBase Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-17234.patch
>
>
> No new development has been done on the HBase metastore in at least a year, 
> and to my knowledge no one is using it (nor is it even in a state to be fully 
> usable).  Given the lack of interest in continuing to develop it, we should 
> remove it rather than leave dead code hanging around and extra tests taking 
> up time in test runs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17234) Remove HBase metastore from master

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261689#comment-16261689
 ] 

ASF GitHub Bot commented on HIVE-17234:
---

Github user alanfgates closed the pull request at:

https://github.com/apache/hive/pull/218


> Remove HBase metastore from master
> --
>
> Key: HIVE-17234
> URL: https://issues.apache.org/jira/browse/HIVE-17234
> Project: Hive
>  Issue Type: Task
>  Components: HBase Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-17234.patch
>
>
> No new development has been done on the HBase metastore in at least a year, 
> and to my knowledge no one is using it (nor is it even in a state to be fully 
> usable).  Given the lack of interest in continuing to develop it, we should 
> remove it rather than leave dead code hanging around and extra tests taking 
> up time in test runs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17980:
--
Attachment: HIVE-17980.patch

> Move HiveMetaStoreClient plus a few remaining classes.
> --
>
> Key: HIVE-17980
> URL: https://issues.apache.org/jira/browse/HIVE-17980
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17980.patch
>
>
> This moves over HiveMetaStoreClient a few more remaining things that haven't 
> been moved yet.  It also cleans up the mess I made of MetaStoreUtils.  I 
> renamed the class in metastore to HiveMetaStoreUtils and left a few methods 
> in there that could be moved.  Everything else has been moved to 
> standalone-metastore/.../MetaStoreUtils.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17980:
--
Status: Patch Available  (was: Open)

> Move HiveMetaStoreClient plus a few remaining classes.
> --
>
> Key: HIVE-17980
> URL: https://issues.apache.org/jira/browse/HIVE-17980
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17980.patch
>
>
> This moves over HiveMetaStoreClient a few more remaining things that haven't 
> been moved yet.  It also cleans up the mess I made of MetaStoreUtils.  I 
> renamed the class in metastore to HiveMetaStoreUtils and left a few methods 
> in there that could be moved.  Everything else has been moved to 
> standalone-metastore/.../MetaStoreUtils.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-17980:
--
Labels: pull-request-available  (was: )

> Move HiveMetaStoreClient plus a few remaining classes.
> --
>
> Key: HIVE-17980
> URL: https://issues.apache.org/jira/browse/HIVE-17980
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
>
> This moves over HiveMetaStoreClient a few more remaining things that haven't 
> been moved yet.  It also cleans up the mess I made of MetaStoreUtils.  I 
> renamed the class in metastore to HiveMetaStoreUtils and left a few methods 
> in there that could be moved.  Everything else has been moved to 
> standalone-metastore/.../MetaStoreUtils.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261686#comment-16261686
 ] 

ASF GitHub Bot commented on HIVE-17980:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/272

HIVE-17980 Moved HiveMetaStoreClient plus a few remaining classes.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17980

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/272.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #272


commit 1f88bf94840e75130482e114a990eac5e9e98b88
Author: Alan Gates 
Date:   2017-09-07T02:00:14Z

HIVE-17980 Moved HiveMetaStoreClient plus a few remaining classes.




> Move HiveMetaStoreClient plus a few remaining classes.
> --
>
> Key: HIVE-17980
> URL: https://issues.apache.org/jira/browse/HIVE-17980
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
>
> This moves over HiveMetaStoreClient a few more remaining things that haven't 
> been moved yet.  It also cleans up the mess I made of MetaStoreUtils.  I 
> renamed the class in metastore to HiveMetaStoreUtils and left a few methods 
> in there that could be moved.  Everything else has been moved to 
> standalone-metastore/.../MetaStoreUtils.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261645#comment-16261645
 ] 

Prasanth Jayachandran commented on HIVE-18076:
--

+1, pending tests

> killquery doesn't actually work for non-trigger WM kills, or the error 
> message is not propagated
> 
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice 
> message like triggers do.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18073) AM may assert when duck count for it is reduced

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-18073:
---

Assignee: Sergey Shelukhin

> AM may assert when duck count for it is reduced
> ---
>
> Key: HIVE-18073
> URL: https://issues.apache.org/jira/browse/HIVE-18073
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> Sometimes it asserts that it doesn't have so many ducks to give away. This 
> should never happen, need to debug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18100) Some tests time out

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261634#comment-16261634
 ] 

Hive QA commented on HIVE-18100:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898570/HIVE-18100.2.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11368 tests 
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=159)

[dynamic_semijoin_reduction.q,vectorization_pushdown.q,correlationoptimizer2.q,cbo_gby_empty.q,resourceplan.q,vectorization_short_regress.q,identity_project_remove_skip.q,mapjoin3.q,cross_product_check_1.q,bucket2.q,unionDistinct_3.q,cbo_join.q,correlationoptimizer6.q,union_remove_26.q,cbo_rp_limit.q,vector_groupby_cube1.q,vectorization_13.q,unionDistinct_1.q,mapjoin_mapjoin.q,union2.q,groupby2.q,vector_decimal_precision.q,dynpart_sort_opt_vectorization.q,exchgpartition2lel.q,sample10.q,vectorized_timestamp_ints_casts.q,auto_sortmerge_join_2.q,bucketizedhiveinputformat.q,cte_mat_2.q,vectorization_8.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_range_multiorder]
 (batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=223)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=229)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=229)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=229)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7947/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7947/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7947/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898570 - PreCommit-HIVE-Build

> Some tests time out
> ---
>
> Key: HIVE-18100
> URL: https://issues.apache.org/jira/browse/HIVE-18100
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-18100.1.patch, HIVE-18100.2.patch, HIVE-18100.patch
>
>
> Some tests had 100s of queries in a single query which times out resulting in 
> Hive QA failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18076:

Status: Patch Available  (was: Open)

> killquery doesn't actually work for non-trigger WM kills, or the error 
> message is not propagated
> 
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice 
> message like triggers do.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18076:

Attachment: HIVE-18076.patch

A small patch; query ID is cleared too early, before the killing is attempted.
[~prasanth_j] can you take a look?

> killquery doesn't actually work for non-trigger WM kills, or the error 
> message is not propagated
> 
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice 
> message like triggers do.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-12360) Bad seek in uncompressed ORC with predicate pushdown

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-12360:
-
   Resolution: Fixed
Fix Version/s: 1.3.0
   Status: Resolved  (was: Patch Available)

Ran some bloom filter and ORC ppd tests and all seems to pass. Committed patch 
to branch-1.

> Bad seek in uncompressed ORC with predicate pushdown
> 
>
> Key: HIVE-12360
> URL: https://issues.apache.org/jira/browse/HIVE-12360
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Hive
>Affects Versions: 1.2.1
> Environment: Oracle Linux 6.4, HDP 2.3.2.0-2950
>Reporter: Gabriel C Balan
>Assignee: Prasanth Jayachandran
> Fix For: 1.3.0
>
> Attachments: HIVE-12360-branch-1.patch, HIVE-12360.1.patch, 
> numtab_100k.csv.gz, orc_test.hive
>
>
> Reading from an ORC file bombs in HDP-2.3.2 when pushing down predicate:
> {noformat:title=Error message in CLI}
> Failed with exception java.io.IOException:java.lang.IllegalArgumentException: 
> Seek in index to 4613 is outside of the data
> {noformat}
> {noformat:title=Stack trace in log4j file}
> 2015-11-06 09:48:11,873 ERROR [main]: CliDriver 
> (SessionState.java:printError(960)) - Failed with exception 
> java.io.IOException:java.lang.IllegalArgumentException: Seek in index to 4613 
> is outside of the data
> java.io.IOException: java.lang.IllegalArgumentException: Seek in index to 
> 4613 is outside of the data
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
>   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalArgumentException: Seek in index to 4613 is 
> outside of the data
>   at 
> org.apache.hadoop.hive.ql.io.orc.InStream$UncompressedStream.seek(InStream.java:139)
>   at 
> org.apache.hadoop.hive.ql.io.orc.InStream$UncompressedStream.read(InStream.java:87)
>   at java.io.InputStream.read(InputStream.java:102)
>   at 
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:737)
>   at 
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
>   at 
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.(OrcProto.java:7429)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.(OrcProto.java:7393)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex$1.parsePartialFrom(OrcProto.java:7482)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex$1.parsePartialFrom(OrcProto.java:7477)
>   at 
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.parseFrom(OrcProto.java:7593)
>   at 
> org.apache.hadoop.hive.ql.io.orc.MetadataReader.readRowIndex(MetadataReader.java:88)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readRowIndex(RecordReaderImpl.java:1166)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readRowIndex(RecordReaderImpl.java:1151)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:750)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:777)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:986)
>   at 
> 

[jira] [Commented] (HIVE-12360) Bad seek in uncompressed ORC with predicate pushdown

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261606#comment-16261606
 ] 

Prasanth Jayachandran commented on HIVE-12360:
--

Ptest is likely broken for branch-1.

> Bad seek in uncompressed ORC with predicate pushdown
> 
>
> Key: HIVE-12360
> URL: https://issues.apache.org/jira/browse/HIVE-12360
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Hive
>Affects Versions: 1.2.1
> Environment: Oracle Linux 6.4, HDP 2.3.2.0-2950
>Reporter: Gabriel C Balan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12360-branch-1.patch, HIVE-12360.1.patch, 
> numtab_100k.csv.gz, orc_test.hive
>
>
> Reading from an ORC file bombs in HDP-2.3.2 when pushing down predicate:
> {noformat:title=Error message in CLI}
> Failed with exception java.io.IOException:java.lang.IllegalArgumentException: 
> Seek in index to 4613 is outside of the data
> {noformat}
> {noformat:title=Stack trace in log4j file}
> 2015-11-06 09:48:11,873 ERROR [main]: CliDriver 
> (SessionState.java:printError(960)) - Failed with exception 
> java.io.IOException:java.lang.IllegalArgumentException: Seek in index to 4613 
> is outside of the data
> java.io.IOException: java.lang.IllegalArgumentException: Seek in index to 
> 4613 is outside of the data
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
>   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalArgumentException: Seek in index to 4613 is 
> outside of the data
>   at 
> org.apache.hadoop.hive.ql.io.orc.InStream$UncompressedStream.seek(InStream.java:139)
>   at 
> org.apache.hadoop.hive.ql.io.orc.InStream$UncompressedStream.read(InStream.java:87)
>   at java.io.InputStream.read(InputStream.java:102)
>   at 
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:737)
>   at 
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
>   at 
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.(OrcProto.java:7429)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.(OrcProto.java:7393)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex$1.parsePartialFrom(OrcProto.java:7482)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex$1.parsePartialFrom(OrcProto.java:7477)
>   at 
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.parseFrom(OrcProto.java:7593)
>   at 
> org.apache.hadoop.hive.ql.io.orc.MetadataReader.readRowIndex(MetadataReader.java:88)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readRowIndex(RecordReaderImpl.java:1166)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readRowIndex(RecordReaderImpl.java:1151)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:750)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:777)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:986)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1019)
>   at 
> 

[jira] [Commented] (HIVE-18122) HCatInputFormat cannot read any data when non-native table has partition columns

[Andrew Olson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261578#comment-16261578
 ] 

Andrew Olson commented on HIVE-18122:
-

Until non-native tables officially support being created with "PARTITIONED BY", 
changing the InitializeInput line:

{noformat}
if (table.getPartitionKeys().size() != 0) {
{noformat}

to:

{noformat}
if (!table.isNonNative() && table.getPartitionKeys().size() != 0) {
{noformat}

seems like a reasonable solution. The data selection filter could then 
presumably be supplied to the storage handler and everything would work as 
expected.

> HCatInputFormat cannot read any data when non-native table has partition 
> columns
> 
>
> Key: HIVE-18122
> URL: https://issues.apache.org/jira/browse/HIVE-18122
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Reporter: Andrew Olson
>
> First, some background info: A non-native table can be created with partition 
> columns defined. However, the existence of partition columns for a non-native 
> table is problematic when using {{HCatInputFormat}}. Nothing disallows the 
> table creation, and the documentation [1] does not mention that non-native 
> tables cannot have partition columns. In fact, it suggests that "PARTITIONED 
> BY" can be specified.
> With such a table definition, for any job using {{HCatInputFormat}} no data 
> can ever be read and the cause is not immediately obvious, only revealed via 
> debugging. The bug stems from the 
> {{org.apache.hive.hcatalog.mapreduce.InitializeInput}} class's logic in the 
> {{getInputJobInfo}} method, where it attempts to identify the partitions to 
> read. With partition columns defined, {{table.getPartitionKeys().size()}} is 
> > 0 so it proceeds to the {{listPartitionsByFilter(...)}} code which will 
> never find any partitions, because partitions cannot be added to a non-native 
> table (HIVE-1223). The returned {{InputJobInfo}} then has an empty 
> {{List}} set rather than taking the "Non partitioned table" path 
> where the table's {{StorageDescriptor}} and parameters are used to build a 
> singleton {{PartInfo}} to use.
> This bug is quite similar to HIVE-18087 although it resides in a different 
> layer of Hive.
> We encountered this using the {{HBaseStorageHandler}}, although I don't 
> believe that's a particularly relevant detail.
> [1] 
> https://cwiki.apache.org/confluence/display/Hive/StorageHandlers#StorageHandlers-DDL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HIVE-18050) LlapServiceDriver shoud split HIVE_AUX_JARS_PATH by ':' instead of ','

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin resolved HIVE-18050.
-
   Resolution: Fixed
Fix Version/s: 3.0.0

Committed to master as a diff because the pull request is against a mirror of 
the main Apache git repo. Thanks for the patch!

> LlapServiceDriver shoud split HIVE_AUX_JARS_PATH by ':' instead of ','
> --
>
> Key: HIVE-18050
> URL: https://issues.apache.org/jira/browse/HIVE-18050
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Clients
>Affects Versions: 2.3.0
>Reporter: Aegeaner
>Assignee: Aegeaner
>  Labels: pull-request-available
> Fix For: 3.0.0
>
>
> LlapServiceDriver shoud split HIVE_AUX_JARS_PATH by ':' instead of ',' , 
> since in hive script the environment variable has been replaced:
> {code:java}
> elif [ "${HIVE_AUX_JARS_PATH}" != "" ]; then 
>   HIVE_AUX_JARS_PATH=`echo $HIVE_AUX_JARS_PATH | sed 's/,/:/g'`
>   if $cygwin; then
>   HIVE_AUX_JARS_PATH=`cygpath -p -w "$HIVE_AUX_JARS_PATH"`
>   HIVE_AUX_JARS_PATH=`echo $HIVE_AUX_JARS_PATH | sed 's/;/,/g'`
>   fi
>   AUX_CLASSPATH=${AUX_CLASSPATH}:${HIVE_AUX_JARS_PATH}
>   AUX_PARAM="file://$(echo ${HIVE_AUX_JARS_PATH} | sed 's/:/,file:\/\//g')"
> fi
> {code}
> But in the LLAP Service Driver, it's processed as :
> {code:java}
>  private void addAuxJarsToSet(HashSet auxJarSet, String auxJars) {
>   if (auxJars != null && !auxJars.isEmpty()) {
> // TODO: transitive dependencies warning?
> String[] jarPaths = auxJars.split(",");
> for (String jarPath : jarPaths) {
>   if (!jarPath.isEmpty()) {
> auxJarSet.add(jarPath);
>   }
> }
>   }
> }
>   };
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18086) NullPointerException initializing query job when non-native table has partition columns

[Andrew Olson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Olson updated HIVE-18086:

Description: 
First, some background info: A non-native table can be created with partition 
columns defined. However, the existence of partition columns for a non-native 
table is problematic. Nothing disallows the table creation, and the 
documentation [1] does not mention that non-native tables cannot have partition 
columns. In fact, it suggests that "PARTITIONED BY" can be specified.

When launching the MR job for any query a {{NullPointerException}} is thrown. A 
null path value originates in 
{{org.apache.hadoop.hive.ql.exec.Utilities#createDummyFileForEmptyTable(...)}} 
because "if this isn't a hive table we can't create an empty file for it" and 
is added to the list of input paths that the 
{{org.apache.hadoop.hive.ql.exec.mr.ExecDriver}} provides to the 
{{FileInputFormat}}, ultimately causing the NPE in the {{Path}} constructor 
because the child path is null.

{noformat}
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.NullPointerException
at org.apache.hadoop.fs.Path.(Path.java:115)
at 
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:438)
at 
org.apache.hadoop.hive.ql.exec.Utilities.setInputPaths(Utilities.java:3550)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:371)
at 
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1690)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1449)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1235)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1053)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:209)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:161)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:372)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:617)
{noformat}

We encountered this bug using the {{HBaseStorageHandler}}.

[1] 
https://cwiki.apache.org/confluence/display/Hive/StorageHandlers#StorageHandlers-DDL


  was:
A non-native table can be created with partition columns defined. However, the 
existence of partition columns for a non-native table is problematic. Nothing 
disallows the table creation, and the documentation [1] does not mention that 
non-native tables cannot have partition columns. In fact, it suggests that 
"PARTITIONED BY" can be specified.

When launching the MR job for any query a NullPointerException is thrown. A 
null path value originates in 
{{org.apache.hadoop.hive.ql.exec.Utilities#createDummyFileForEmptyTable(...)}} 
because "if this isn't a hive table we can't create an empty file for it" and 
is added to the list of input paths that the 
{{org.apache.hadoop.hive.ql.exec.mr.ExecDriver}} provides to the 
FileInputFormat, ultimately causing the NPE in the Path constructor because the 
child path is null.

{noformat}
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.NullPointerException
at org.apache.hadoop.fs.Path.(Path.java:115)
at 
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:438)
at 
org.apache.hadoop.hive.ql.exec.Utilities.setInputPaths(Utilities.java:3550)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:371)
at 
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1690)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1449)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1235)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1053)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:209)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:161)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:372)
at 

[jira] [Updated] (HIVE-18087) Simple select query finds nothing when non-native table has partition columns

[Andrew Olson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Olson updated HIVE-18087:

Description: 
First, some background info: A non-native table can be created with partition 
columns defined. However, the existence of partition columns for a non-native 
table is problematic. Nothing disallows the table creation, and the 
documentation [1] does not mention that non-native tables cannot have partition 
columns. In fact, it suggests that "PARTITIONED BY" can be specified.

With this table definition, when running a query that does not launch a MR job, 
the query completes successfully and immediately, but no rows are ever 
returned. I have not yet been able to identify the specific code logic path 
that prevents it from working as expected, perhaps something in the query plan 
related to partition pruning.

This bug is similar to HIVE-18086. We see this issue with a query like "select 
\* from my_table;" and the HIVE-18086 {{NullPointerException}} with a query 
like "select count(\*) from my_table;".

We encountered this bug using the {{HBaseStorageHandler}}. However, enabling 
debug logging suggests that the query is being short-circuited without actually 
connecting to HBase at all.

[1] 
https://cwiki.apache.org/confluence/display/Hive/StorageHandlers#StorageHandlers-DDL

  was:
A non-native table can be created with partition columns defined. However, the 
existence of partition columns for a non-native table is problematic. Nothing 
disallows the table creation, and the documentation [1] does not mention that 
non-native tables cannot have partition columns. In fact, it suggests that 
"PARTITIONED BY" can be specified.

With this table definition, when running a query that does not launch a MR job, 
the query completes successfully and immediately, but no rows are ever 
returned. I have not yet been able to identify the specific code logic path 
that prevents it from working as expected, perhaps something in the query plan 
related to partition pruning.

This bug is similar to HIVE-18086. We see this issue with a query like "select 
\* from my_table;" and the HIVE-18086 NullPointerException with a query like 
"select count(\*) from my_table;".

We encountered this bug using the HBaseStorageHandler. However, enabling debug 
logging suggests that the query is being short-circuited without actually 
connecting to HBase at all.

[1] 
https://cwiki.apache.org/confluence/display/Hive/StorageHandlers#StorageHandlers-DDL


> Simple select query finds nothing when non-native table has partition columns
> -
>
> Key: HIVE-18087
> URL: https://issues.apache.org/jira/browse/HIVE-18087
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
> Environment: CDH 5.5.5 on RHEL 6.8
>Reporter: Andrew Olson
>
> First, some background info: A non-native table can be created with partition 
> columns defined. However, the existence of partition columns for a non-native 
> table is problematic. Nothing disallows the table creation, and the 
> documentation [1] does not mention that non-native tables cannot have 
> partition columns. In fact, it suggests that "PARTITIONED BY" can be 
> specified.
> With this table definition, when running a query that does not launch a MR 
> job, the query completes successfully and immediately, but no rows are ever 
> returned. I have not yet been able to identify the specific code logic path 
> that prevents it from working as expected, perhaps something in the query 
> plan related to partition pruning.
> This bug is similar to HIVE-18086. We see this issue with a query like 
> "select \* from my_table;" and the HIVE-18086 {{NullPointerException}} with a 
> query like "select count(\*) from my_table;".
> We encountered this bug using the {{HBaseStorageHandler}}. However, enabling 
> debug logging suggests that the query is being short-circuited without 
> actually connecting to HBase at all.
> [1] 
> https://cwiki.apache.org/confluence/display/Hive/StorageHandlers#StorageHandlers-DDL



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-12360) Bad seek in uncompressed ORC with predicate pushdown

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261533#comment-16261533
 ] 

Sergey Shelukhin commented on HIVE-12360:
-

+1

> Bad seek in uncompressed ORC with predicate pushdown
> 
>
> Key: HIVE-12360
> URL: https://issues.apache.org/jira/browse/HIVE-12360
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Hive
>Affects Versions: 1.2.1
> Environment: Oracle Linux 6.4, HDP 2.3.2.0-2950
>Reporter: Gabriel C Balan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12360-branch-1.patch, HIVE-12360.1.patch, 
> numtab_100k.csv.gz, orc_test.hive
>
>
> Reading from an ORC file bombs in HDP-2.3.2 when pushing down predicate:
> {noformat:title=Error message in CLI}
> Failed with exception java.io.IOException:java.lang.IllegalArgumentException: 
> Seek in index to 4613 is outside of the data
> {noformat}
> {noformat:title=Stack trace in log4j file}
> 2015-11-06 09:48:11,873 ERROR [main]: CliDriver 
> (SessionState.java:printError(960)) - Failed with exception 
> java.io.IOException:java.lang.IllegalArgumentException: Seek in index to 4613 
> is outside of the data
> java.io.IOException: java.lang.IllegalArgumentException: Seek in index to 
> 4613 is outside of the data
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
>   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalArgumentException: Seek in index to 4613 is 
> outside of the data
>   at 
> org.apache.hadoop.hive.ql.io.orc.InStream$UncompressedStream.seek(InStream.java:139)
>   at 
> org.apache.hadoop.hive.ql.io.orc.InStream$UncompressedStream.read(InStream.java:87)
>   at java.io.InputStream.read(InputStream.java:102)
>   at 
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:737)
>   at 
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
>   at 
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.(OrcProto.java:7429)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.(OrcProto.java:7393)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex$1.parsePartialFrom(OrcProto.java:7482)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex$1.parsePartialFrom(OrcProto.java:7477)
>   at 
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
>   at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex.parseFrom(OrcProto.java:7593)
>   at 
> org.apache.hadoop.hive.ql.io.orc.MetadataReader.readRowIndex(MetadataReader.java:88)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readRowIndex(RecordReaderImpl.java:1166)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readRowIndex(RecordReaderImpl.java:1151)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:750)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:777)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:986)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1019)
>   at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:205)
>   at 
> 

[jira] [Updated] (HIVE-14487) Add REBUILD statement for materialized views

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14487:
---
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master, thanks for reviewing [~ashutoshc]!

> Add REBUILD statement for materialized views
> 
>
> Key: HIVE-14487
> URL: https://issues.apache.org/jira/browse/HIVE-14487
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 3.0.0
>
> Attachments: HIVE-14487.01.patch, HIVE-14487.02.patch, 
> HIVE-14487.03.patch, HIVE-14487.04.patch, HIVE-14487.patch
>
>
> Support for rebuilding existing materialized views. The statement is the 
> following:
> {code:sql}
> ALTER MATERIALIZED VIEW [db_name.]materialized_view_name REBUILD;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18115) Fix schema version info for Hive-2.3.2

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261531#comment-16261531
 ] 

Sahil Takiar commented on HIVE-18115:
-

2.3.2 has already been released. So I guess this will have to go into 2.3.3

> Fix schema version info for Hive-2.3.2
> --
>
> Key: HIVE-18115
> URL: https://issues.apache.org/jira/browse/HIVE-18115
> Project: Hive
>  Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Fix For: 2.3.2
>
> Attachments: HIVE-18115.1.patch
>
>
> Error while starting HiveMeta
> {code}
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema 
> version 2.3.2 does not match metastore's schema version 2.3.0 Metastore is 
> not upgraded or corrupt
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7600)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7563)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101) 
> ~[hive-exec-2.3.2.jar:2.3.2]
> at com.sun.proxy.$Proxy23.verifySchema(Unknown Source) ~[?:?]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18109) Don't reserve pool and default as keywords

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261529#comment-16261529
 ] 

Ashutosh Chauhan commented on HIVE-18109:
-

Please take a look at TestCliDriver.testCliDriver[dbtxnmgr_showlocks] This test 
started failing after HIVE-17902 commit and is failing since for every run.

> Don't reserve pool and default as keywords
> --
>
> Key: HIVE-18109
> URL: https://issues.apache.org/jira/browse/HIVE-18109
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
>
> HIVE-17902 broke this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18037) Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261527#comment-16261527
 ] 

Sergey Shelukhin commented on HIVE-18037:
-

[~gsaha] looks like the desired package is not found in the repo. Not sure why 
it's going to DN repo, presumably it was the last one/first one that it tried

> Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x
> 
>
> Key: HIVE-18037
> URL: https://issues.apache.org/jira/browse/HIVE-18037
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Gour Saha
> Fix For: 3.0.0
>
> Attachments: HIVE-18037.001.patch
>
>
> Apache Slider has been migrated to Hadoop-3.x and is referred to as YARN 
> Service (YARN-4692). Most of the classic Slider features are now going to be 
> supported in a first-class manner by core YARN. It includes several new 
> features like a RESTful API. Command line equivalents of classic Slider are 
> supported by YARN Service as well.
> This jira will take care of all changes required to Slider LLAP packaging and 
> scripts to make it work against Hadoop 3.x.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17937) llap_acid_fast test is flaky

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261524#comment-16261524
 ] 

Sergey Shelukhin commented on HIVE-17937:
-

[~teddy.choi] do you have any update? thanks

> llap_acid_fast test is flaky
> 
>
> Key: HIVE-17937
> URL: https://issues.apache.org/jira/browse/HIVE-17937
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Teddy Choi
> Attachments: HIVE-17993.patch
>
>
> See for example 
> https://builds.apache.org/job/PreCommit-HIVE-Build/7521/testReport/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_llap_acid_fast_/history/
>  (the history link is the same from any build number with a test run, just 
> replace 7521 if this one expires).
> Looks like results change, which may not be good.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18109) Don't reserve pool and default as keywords

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261522#comment-16261522
 ] 

Sergey Shelukhin commented on HIVE-18109:
-

[~ashutoshc] can you elaborate? they are already added to the non-reserved list

> Don't reserve pool and default as keywords
> --
>
> Key: HIVE-18109
> URL: https://issues.apache.org/jira/browse/HIVE-18109
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
>
> HIVE-17902 broke this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18121) TPCDS query 19 runs throws NPE in VectorizedParquetRecordReader initializing tables partition info

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261519#comment-16261519
 ] 

Sergey Shelukhin commented on HIVE-18121:
-

It doesn't look related to cache in particular... can you try with that fix? or 
disable cache.

> TPCDS query 19 runs throws NPE in VectorizedParquetRecordReader initializing 
> tables partition info 
> ---
>
> Key: HIVE-18121
> URL: https://issues.apache.org/jira/browse/HIVE-18121
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, llap
>Affects Versions: 3.0.0
>Reporter: Nita Dembla
>  Labels: parquet
>
> Testing TPCDS 1TB with LLAP Parquet cache. Ran into the following exception 
> {code}
> 2017-11-21T00:53:33,605 ERROR [HiveServer2-Background-Pool: Thread-330] 
> ql.Driver: FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, 
> vertexId=vertex_1509557139747_0295_48_02, diagnostics=[Task failed, 
> taskId=task_1509557139747_0295_48_02_000105, diagnostics=[TaskAttempt 0 
> failed, info=[Error: Error while running task ( failure ) : 
> attempt_1509557139747_0295_48_02_000105_0:java.lang.RuntimeException: 
> java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at 
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.io.IOException: 
> java.lang.RuntimeException: java.lang.NullPointerException
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at 
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
> at 
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
> at 
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at 
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at 
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at 
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:525)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:171)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:253)
> ... 15 more
> Caused by: java.io.IOException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at 
> 

[jira] [Commented] (HIVE-15102) Hiveptest is killing nodes where IP is reused after previous node termination

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261509#comment-16261509
 ] 

Ashutosh Chauhan commented on HIVE-15102:
-

In some of the recent test runs some batches are timing out (ofcourse randomly 
but rarely). I looked into log of one such failure and found it contains 
following:
{code}
2017-11-18T10:30:01,231  WARN [Fetcher_O {Map_1} #0] 
orderedgrouped.FetcherOrderedGrouped: Failed to connect to 
hive-ptest-slaves-aff.c.gcp-hive-upstream.internal:0 with 1 inputs
java.io.IOException: Failed to connect to 
http://hive-ptest-slaves-aff.c.gcp-hive-upstream.internal:0/mapOutput?job=job_1511029513075_0001=203=0=attempt_1511029513075_0001_203_00_00_0_11576,
 #connectionFailures=3
at org.apache.tez.http.HttpConnection.connect(HttpConnection.java:168) 
~[tez-runtime-library-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT]
{code}

Above suggested to me that some slaves went away in middle of test execution 
resulting in those time outs.

> Hiveptest is killing nodes where IP is reused after previous node termination
> -
>
> Key: HIVE-15102
> URL: https://issues.apache.org/jira/browse/HIVE-15102
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-15102.1.patch
>
>
> NO PRECOMMIT TESTS
> The Hiveptest framework has a background thread that runs every hour, and 
> attempts to kill zombie nodes that are not being used by the test execution 
> anymore. 
> These killed nodes are kept in a list of terminated nodes, and next time the 
> background thread is executed, it will attempt to kill all those nodes again 
> because Hiveptest consider them as zombie nodes.
> The problem is that cloud providers can give you the same IP numbers for new 
> nodes, and when the background thread runs, it will kill those nodes that may 
> still be in used by Hiveptest.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14487) Add REBUILD statement for materialized views

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261480#comment-16261480
 ] 

Hive QA commented on HIVE-14487:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898715/HIVE-14487.04.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11399 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7946/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7946/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7946/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12898715 - PreCommit-HIVE-Build

> Add REBUILD statement for materialized views
> 
>
> Key: HIVE-14487
> URL: https://issues.apache.org/jira/browse/HIVE-14487
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14487.01.patch, HIVE-14487.02.patch, 
> HIVE-14487.03.patch, HIVE-14487.04.patch, HIVE-14487.patch
>
>
> Support for rebuilding existing materialized views. The statement is the 
> following:
> {code:sql}
> ALTER MATERIALIZED VIEW [db_name.]materialized_view_name REBUILD;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18052) Run p-tests on mm tables

[Steve Yeom (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261459#comment-16261459
 ] 

Steve Yeom commented on HIVE-18052:
---

Hey Sergey.. I understand your point. Any input is welcome at this point. 

Checking the whole picture before taking actions.

> Run p-tests on mm tables
> 
>
> Key: HIVE-18052
> URL: https://issues.apache.org/jira/browse/HIVE-18052
> Project: Hive
>  Issue Type: Task
>Reporter: Steve Yeom
> Attachments: HIVE-18052.1.patch, HIVE-18052.2.patch, 
> HIVE-18052.3.patch, HIVE-18052.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18115) Fix schema version info for Hive-2.3.2

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261432#comment-16261432
 ] 

Vihang Karajgaonkar commented on HIVE-18115:


[~ashutoshc] yes, you are right. {{.put("2.3.1", "2.3.0") }} should be included 
as well. So technically this affects 2.3.1 version as well. [~osayankin] Can 
you reattach the patch with the suggested fixes. Please name the patch with 
format .-branch-2.patch so that the precommit tests 
run on that branch. So in your case the next patch should be named 
{{HIVE-18115.02-branch-2.patch}}

> Fix schema version info for Hive-2.3.2
> --
>
> Key: HIVE-18115
> URL: https://issues.apache.org/jira/browse/HIVE-18115
> Project: Hive
>  Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Fix For: 2.3.2
>
> Attachments: HIVE-18115.1.patch
>
>
> Error while starting HiveMeta
> {code}
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema 
> version 2.3.2 does not match metastore's schema version 2.3.0 Metastore is 
> not upgraded or corrupt
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7600)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7563)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101) 
> ~[hive-exec-2.3.2.jar:2.3.2]
> at com.sun.proxy.$Proxy23.verifySchema(Unknown Source) ~[?:?]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261431#comment-16261431
 ] 

Sergey Shelukhin commented on HIVE-17954:
-

Mostly looks good; a couple minor comments.
There are a couple comments remaining for tests - adding tests for nested pools 
with rename and drop, dropping a pool that is set as default.

> Implement pool, user, group and trigger to pool management API's.
> -
>
> Key: HIVE-17954
> URL: https://issues.apache.org/jira/browse/HIVE-17954
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-17954.01.patch, HIVE-17954.02.patch, 
> HIVE-17954.03.patch, HIVE-17954.04.patch, HIVE-17954.05.patch, 
> HIVE-17954.06.patch, HIVE-17954.07.patch, HIVE-17954.08.patch
>
>
> Implement the following commands:
> -- Pool management.
> CREATE POOL `resource_plan`.`pool_path` WITH
>   ALLOC_FRACTION `fraction`
>   QUERY_PARALLELISM `parallelism`
>   SCHEDULING_POLICY `policy`;
> ALTER POOL `resource_plan`.`pool_path` SET
>   PATH = `new_path`,
>   ALLOC_FRACTION = `fraction`,
>   QUERY_PARALLELISM = `parallelism`,
>   SCHEDULING_POLICY = `policy`;
> DROP POOL `resource_plan`.`pool_path`;
> -- Trigger to pool mappings.
> ALTER RESOURCE PLAN `resource_plan`
>   ADD TRIGGER `trigger_name` TO `pool_path`;
> ALTER RESOURCE PLAN `resource_plan`
>   DROP TRIGGER `trigger_name` TO `pool_path`;
> -- User/Group to pool mappings.
> CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name`
>   TO `pool_path` WITH ORDERING `order_no`;
> DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17967) Move HiveMetaStore class

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17967:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Patch 4 committed.  Thank you Thejas for the feedback and review.

> Move HiveMetaStore class
> 
>
> Key: HIVE-17967
> URL: https://issues.apache.org/jira/browse/HIVE-17967
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-17967.2.patch, HIVE-17967.3.patch, 
> HIVE-17967.4.patch, HIVE-17967.patch
>
>
> We need to move HiveMetaStore and a few tightly integrated classes into the 
> standalone metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17967) Move HiveMetaStore class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261423#comment-16261423
 ] 

ASF GitHub Bot commented on HIVE-17967:
---

Github user asfgit closed the pull request at:

https://github.com/apache/hive/pull/270


> Move HiveMetaStore class
> 
>
> Key: HIVE-17967
> URL: https://issues.apache.org/jira/browse/HIVE-17967
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17967.2.patch, HIVE-17967.3.patch, 
> HIVE-17967.4.patch, HIVE-17967.patch
>
>
> We need to move HiveMetaStore and a few tightly integrated classes into the 
> standalone metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18105) Aggregation of an empty set doesn't pass constants to the UDAF

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261419#comment-16261419
 ] 

Ashutosh Chauhan commented on HIVE-18105:
-

+1

> Aggregation of an empty set doesn't pass constants to the UDAF
> --
>
> Key: HIVE-18105
> URL: https://issues.apache.org/jira/browse/HIVE-18105
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18105.01.patch, HIVE-18105.02.patch, 
> HIVE-18105.02.patch
>
>
> the groupbyoperator's logic for firstrow passes {{null}} for all parameters.
> see 
> [here|https://github.com/apache/hive/blob/39d46e8af5a3794f7395060b890f94ddc84516e7/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java#L1116].
> this could obstruct {{compute_stats}} operations because it has a constant 
> argument.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18052) Run p-tests on mm tables

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261421#comment-16261421
 ] 

Sergey Shelukhin commented on HIVE-18052:
-

[~steveyeom2017] it looks like most of the tests failed with trivial diffs of 
table type (all the explains say "transactional true", etc.). 
Is there reason you want to disable the src table being MM? It probably won't 
fix those, at least for other tables.  Maybe again they can be removed from 
output if the table is MM, instead (for this exercise).


> Run p-tests on mm tables
> 
>
> Key: HIVE-18052
> URL: https://issues.apache.org/jira/browse/HIVE-18052
> Project: Hive
>  Issue Type: Task
>Reporter: Steve Yeom
> Attachments: HIVE-18052.1.patch, HIVE-18052.2.patch, 
> HIVE-18052.3.patch, HIVE-18052.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18115) Fix schema version info for Hive-2.3.2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261399#comment-16261399
 ] 

Hive QA commented on HIVE-18115:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898698/HIVE-18115.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 11387 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_2]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_3]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_4]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_5]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_6]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_7]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_create_db]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_drop_db]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_1]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_2]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_3]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_4]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_5]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_6]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_7]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_1]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_2]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_3]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_4]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_5]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_6]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_7]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_22_export_authfail]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_23_import_exist_authfail]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_24_import_part_authfail]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_25_import_nonexist_authfail]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_exist_part_authfail]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_nonpart_authfail]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_part_authfail]
 (batchId=92)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=223)
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
 (batchId=225)
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.testReadDataFromEncryptedHiveTableByPig[0]
 (batchId=187)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=229)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=229)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=229)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7945/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7945/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7945/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 39 

[jira] [Commented] (HIVE-18112) show create for view having special char in where clause is not showing properly

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261391#comment-16261391
 ] 

Ashutosh Chauhan commented on HIVE-18112:
-

[~nareshpr] Ususally practice is to get patch first committed to master, which 
implies you need to generate patch against master branch and have it tested, 
reviewed and committed.  
Once that is done, that you may generate patch for a specific branch and attach 
it here which can then be committed. For that see: 
https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing

> show create for view having special char in where clause is not showing 
> properly
> 
>
> Key: HIVE-18112
> URL: https://issues.apache.org/jira/browse/HIVE-18112
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HIVE-18112.patch
>
>
> e.g., 
> CREATE VIEW `v2` AS select `evil_byte1`.`a` from `default`.`EVIL_BYTE1` where 
> `evil_byte1`.`a` = 'abcÖdefÖgh';
> Output:
> ==
> 0: jdbc:hive2://172.26.122.227:1> show create table v2;
> ++--+
> | createtab_stmt  
>|
> ++--+
> | CREATE VIEW `v2` AS select `evil_byte1`.`a` from `default`.`EVIL_BYTE1` 
> where `evil_byte1`.`a` = 'abc�def�gh'  |
> ++--+
> Only show create output is having invalid characters, actual source table 
> content is displayed properly in the console.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18115) Fix schema version info for Hive-2.3.2

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261381#comment-16261381
 ] 

Ashutosh Chauhan commented on HIVE-18115:
-

Shall this also include :  .put("2.3.1", "2.3.0") ?

> Fix schema version info for Hive-2.3.2
> --
>
> Key: HIVE-18115
> URL: https://issues.apache.org/jira/browse/HIVE-18115
> Project: Hive
>  Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Fix For: 2.3.2
>
> Attachments: HIVE-18115.1.patch
>
>
> Error while starting HiveMeta
> {code}
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema 
> version 2.3.2 does not match metastore's schema version 2.3.0 Metastore is 
> not upgraded or corrupt
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7600)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7563)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101) 
> ~[hive-exec-2.3.2.jar:2.3.2]
> at com.sun.proxy.$Proxy23.verifySchema(Unknown Source) ~[?:?]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_141]
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
>  ~[hive-exec-2.3.2.jar:2.3.2]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14487) Add REBUILD statement for materialized views

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14487:
---
Attachment: HIVE-14487.04.patch

> Add REBUILD statement for materialized views
> 
>
> Key: HIVE-14487
> URL: https://issues.apache.org/jira/browse/HIVE-14487
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14487.01.patch, HIVE-14487.02.patch, 
> HIVE-14487.03.patch, HIVE-14487.04.patch, HIVE-14487.patch
>
>
> Support for rebuilding existing materialized views. The statement is the 
> following:
> {code:sql}
> ALTER MATERIALIZED VIEW [db_name.]materialized_view_name REBUILD;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (HIVE-18063) Make CommandProcessorResponse an exception instead of a return class

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich reopened HIVE-18063:
-

reverted; broke a lot of tests...I might have applied the wrong patch?

> Make CommandProcessorResponse an exception instead of a return class
> 
>
> Key: HIVE-18063
> URL: https://issues.apache.org/jira/browse/HIVE-18063
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Fix For: 3.0.0
>
> Attachments: HIVE-17063.02.patch, HIVE-18063.01.patch
>
>
> the usage pattern of the {{CommandProcessorResponse}} class suggests that its 
> current role is closer to Exceptions than to return values.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18113) Remove mixed partitions/table schema support

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261282#comment-16261282
 ] 

Ashutosh Chauhan commented on HIVE-18113:
-

I am not sure what you are proposing. From patch it seems like you are removing 
{{cascade}} support in alter statement, though description and title of jira 
convey differently.
In any case, capability of table and partition having different schema is one 
of key features of Hive which we don't want to remove unless there is a strong 
justification for it.

> Remove mixed partitions/table schema support
> 
>
> Key: HIVE-18113
> URL: https://issues.apache.org/jira/browse/HIVE-18113
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18113.01.patch
>
>
> Currently it is possible to have misaligned table/partition schemas;
> [see here for example| 
> https://github.com/apache/hive/blob/a5c2e15c7cc125d8cda2ee3a8ed64c116ff6b755/ql/src/test/queries/clientpositive/schema_evol_text_vec_part.q#L156]
> result of [this insert 
> statement|https://github.com/apache/hive/blob/a5c2e15c7cc125d8cda2ee3a8ed64c116ff6b755/ql/src/test/queries/clientpositive/schema_evol_text_vec_part.q#L162]
>  are these [null 
> values|https://github.com/apache/hive/blob/a5c2e15c7cc125d8cda2ee3a8ed64c116ff6b755/ql/src/test/results/clientpositive/llap/schema_evol_text_vec_part.q.out#L660]
> This mixed partition setup can cause the stats aggregation to become quite 
> confusing...I think it would be better to remove this thingthere is a 
> {{CASCADE}} flag already; which changes the schema all over the 
> table/partitions/etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18121) TPCDS query 19 runs throws NPE in VectorizedParquetRecordReader initializing tables partition info

[Nita Dembla (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261275#comment-16261275
 ] 

Nita Dembla commented on HIVE-18121:


[~vihangk1] Version - Hive 3.0. I don't have the fix for HIVE-17961 in my 
build. 

> TPCDS query 19 runs throws NPE in VectorizedParquetRecordReader initializing 
> tables partition info 
> ---
>
> Key: HIVE-18121
> URL: https://issues.apache.org/jira/browse/HIVE-18121
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, llap
>Affects Versions: 3.0.0
>Reporter: Nita Dembla
>  Labels: parquet
>
> Testing TPCDS 1TB with LLAP Parquet cache. Ran into the following exception 
> {code}
> 2017-11-21T00:53:33,605 ERROR [HiveServer2-Background-Pool: Thread-330] 
> ql.Driver: FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, 
> vertexId=vertex_1509557139747_0295_48_02, diagnostics=[Task failed, 
> taskId=task_1509557139747_0295_48_02_000105, diagnostics=[TaskAttempt 0 
> failed, info=[Error: Error while running task ( failure ) : 
> attempt_1509557139747_0295_48_02_000105_0:java.lang.RuntimeException: 
> java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at 
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.io.IOException: 
> java.lang.RuntimeException: java.lang.NullPointerException
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at 
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
> at 
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
> at 
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at 
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at 
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at 
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:525)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:171)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:253)
> ... 15 more
> Caused by: java.io.IOException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at 
> 

[jira] [Updated] (HIVE-18121) TPCDS query 19 runs throws NPE in VectorizedParquetRecordReader initializing tables partition info

[Nita Dembla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nita Dembla updated HIVE-18121:
---
Affects Version/s: 3.0.0

> TPCDS query 19 runs throws NPE in VectorizedParquetRecordReader initializing 
> tables partition info 
> ---
>
> Key: HIVE-18121
> URL: https://issues.apache.org/jira/browse/HIVE-18121
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, llap
>Affects Versions: 3.0.0
>Reporter: Nita Dembla
>  Labels: parquet
>
> Testing TPCDS 1TB with LLAP Parquet cache. Ran into the following exception 
> {code}
> 2017-11-21T00:53:33,605 ERROR [HiveServer2-Background-Pool: Thread-330] 
> ql.Driver: FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, 
> vertexId=vertex_1509557139747_0295_48_02, diagnostics=[Task failed, 
> taskId=task_1509557139747_0295_48_02_000105, diagnostics=[TaskAttempt 0 
> failed, info=[Error: Error while running task ( failure ) : 
> attempt_1509557139747_0295_48_02_000105_0:java.lang.RuntimeException: 
> java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at 
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.io.IOException: 
> java.lang.RuntimeException: java.lang.NullPointerException
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at 
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
> at 
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
> at 
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at 
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at 
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at 
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:525)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:171)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:253)
> ... 15 more
> Caused by: java.io.IOException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:418)
> at 
> 

[jira] [Commented] (HIVE-14487) Add REBUILD statement for materialized views

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261269#comment-16261269
 ] 

Hive QA commented on HIVE-14487:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898687/HIVE-14487.03.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 41 failed/errored test(s), 11399 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] 
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] 
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=165)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_2]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_3]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_4]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_5]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_6]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_7]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_create_db]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_fail_drop_db]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_1]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_2]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_3]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_4]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_5]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_6]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_7]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_1]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_2]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_3]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_4]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_5]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_6]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_view_disable_cbo_7]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_view_failure1]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_22_export_authfail]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_23_import_exist_authfail]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_24_import_part_authfail]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_25_import_nonexist_authfail]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_exist_part_authfail]
 (batchId=91)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_nonpart_authfail]
 (batchId=92)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[load_part_authfail]
 (batchId=92)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=224)
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
 (batchId=226)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7944/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7944/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7944/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing 

[jira] [Commented] (HIVE-18121) TPCDS query 19 runs throws NPE in VectorizedParquetRecordReader initializing tables partition info

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261265#comment-16261265
 ] 

Vihang Karajgaonkar commented on HIVE-18121:


This may be duplicate of HIVE-17961 or HIVE-15718. Hi [~ndembla] Can you 
specify the version where you see this exception?

> TPCDS query 19 runs throws NPE in VectorizedParquetRecordReader initializing 
> tables partition info 
> ---
>
> Key: HIVE-18121
> URL: https://issues.apache.org/jira/browse/HIVE-18121
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, llap
>Reporter: Nita Dembla
>  Labels: parquet
>
> Testing TPCDS 1TB with LLAP Parquet cache. Ran into the following exception 
> {code}
> 2017-11-21T00:53:33,605 ERROR [HiveServer2-Background-Pool: Thread-330] 
> ql.Driver: FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, 
> vertexId=vertex_1509557139747_0295_48_02, diagnostics=[Task failed, 
> taskId=task_1509557139747_0295_48_02_000105, diagnostics=[TaskAttempt 0 
> failed, info=[Error: Error while running task ( failure ) : 
> attempt_1509557139747_0295_48_02_000105_0:java.lang.RuntimeException: 
> java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at 
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.io.IOException: 
> java.lang.RuntimeException: java.lang.NullPointerException
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at 
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at 
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:157)
> at 
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:83)
> at 
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at 
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at 
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at 
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:525)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:171)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:253)
> ... 15 more
> Caused by: java.io.IOException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at 
>