date:20181026

[jira] [Created] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

2018-10-26 Thread Tom Zeng (JIRA)

Tom Zeng created HIVE-20825:
---

 Summary: Hive ACID Merge generates invalid ORC files (bucket files 
0 or 3 bytes in length) causing the "Not a valid ORC file" error
 Key: HIVE-20825
 URL: https://issues.apache.org/jira/browse/HIVE-20825
 Project: Hive
  Issue Type: Bug
  Components: Hive, ORC, Transactions
Affects Versions: 2.3.2, 2.3.1, 2.2.0
 Environment: Hive 2.3.x on Amazon EMR 5.8.0 to 5.18.0
Reporter: Tom Zeng


When using Hive ACID Merge (supported with the ORC format) to update/insert 
data, bucket files with 0 byte or 3 bytes (file content is three character: 
ORC) are generated during MERGE INTO operations which finish with no errors. 
Subsequent queries on the base table will get "Not a valid ORC file" error.

 

The following script can be used to reproduce the issue:

set hive.auto.convert.join=false;
set hive.enforce.bucketing=true;
set hive.exec.dynamic.partition.mode = nonstrict;
set hive.support.concurrency=true;
set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

drop table if exists mergedelta_txt_1;
drop table if exists mergedelta_txt_2;

CREATE TABLE mergedelta_txt_1 (
id_str varchar(12), time_key int, value bigint)
PARTITIONED BY (date_key int)
ROW FORMAT DELIMITED
STORED AS TEXTFILE;

CREATE TABLE mergedelta_txt_2 (
id_str varchar(12), time_key int, value bigint)
PARTITIONED BY (date_key int)
ROW FORMAT DELIMITED
STORED AS TEXTFILE;

INSERT INTO TABLE mergedelta_txt_1
partition(date_key=20170103)
VALUES
 ("AB94LIENR0",46700,12345676836978),
 ("AB94LIENR1",46825,12345676836978),
 ("AB94LIENS0",46709,12345676836978),
 ("AB94LIENS1",46834,12345676836978),
 ("AB94LIENT0",46709,12345676836978),
 ("AB94LIENT1",46834,12345676836978),
 ("AB94LIENU0",46718,12345676836978),
 ("AB94LIENU1",46844,12345676836978),
 ("AB94LIENV0",46719,12345676836978),
 ("AB94LIENV1",46844,12345676836978),
 ("AB94LIENW0",46728,12345676836978),
 ("AB94LIENW1",46854,12345676836978),
 ("AB94LIENX0",46728,12345676836978),
 ("AB94LIENX1",46854,12345676836978),
 ("AB94LIENY0",46737,12345676836978),
 ("AB94LIENY1",46863,12345676836978),
 ("AB94LIENZ0",46738,12345676836978),
 ("AB94LIENZ1",46863,12345676836978),
 ("AB94LIERA0",47176,12345676836982),
 ("AB94LIERA1",47302,12345676836982);

INSERT INTO TABLE mergedelta_txt_2
partition(date_key=20170103)
VALUES 
 ("AB94LIENT1",46834,12345676836978),
 ("AB94LIENU0",46718,12345676836978),
 ("AB94LIENU1",46844,12345676836978),
 ("AB94LIENV0",46719,12345676836978),
 ("AB94LIENV1",46844,12345676836978),
 ("AB94LIENW0",46728,12345676836978),
 ("AB94LIENW1",46854,12345676836978),
 ("AB94LIENX0",46728,12345676836978),
 ("AB94LIENX1",46854,12345676836978),
 ("AB94LIENY0",46737,12345676836978),
 ("AB94LIENY1",46863,12345676836978),
 ("AB94LIENZ0",46738,12345676836978),
 ("AB94LIENZ1",46863,12345676836978),
 ("AB94LIERA0",47176,12345676836982),
 ("AB94LIERA1",47302,12345676836982),
 ("AB94LIERA2",47418,12345676836982),
 ("AB94LIERB0",47176,12345676836982),
 ("AB94LIERB1",47302,12345676836982),
 ("AB94LIERB2",47418,12345676836982),
 ("AB94LIERC0",47185,12345676836982);

DROP TABLE IF EXISTS mergebase_1;
CREATE TABLE mergebase_1 (
id_str varchar(12) , time_key int , value bigint)
PARTITIONED BY (date_key int)
CLUSTERED BY (id_str,time_key) INTO 32 BUCKETS
STORED AS ORC
TBLPROPERTIES (
 'orc.compress'='SNAPPY',
 'pk_columns'='id_str,date_key,time_key',
 'NO_AUTO_COMPACTION'='true',
 'transactional'='true');

MERGE INTO mergebase_1 AS base
USING (SELECT * 
 FROM (
 SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
 FROM mergedelta_txt_1
 DISTRIBUTE BY date_key
 ) rankedtbl 
 WHERE rankedtbl.rk=1
) AS delta
ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
delta.time_key=base.time_key
WHEN MATCHED THEN UPDATE SET value=delta.value
WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
delta.value, delta.date_key);

MERGE INTO mergebase_1 AS base
USING (SELECT * 
 FROM (
 SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
 FROM mergedelta_txt_2
 DISTRIBUTE BY date_key
 ) rankedtbl 
 WHERE rankedtbl.rk=1
) AS delta
ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
delta.time_key=base.time_key
WHEN MATCHED THEN UPDATE SET value=delta.value
WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
delta.value, delta.date_key);

select count(*) from mergebase_1;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20824) HiveServer2 is still leaking file handles when using ADD JAR

2018-10-26 Thread hezhang (JIRA)

hezhang created HIVE-20824:
--

 Summary: HiveServer2 is still leaking file handles when using ADD 
JAR 
 Key: HIVE-20824
 URL: https://issues.apache.org/jira/browse/HIVE-20824
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 2.1.1
Reporter: hezhang


In Hive version 2.1.1, hiveserver2 is stll leaking jar file handle when using 
add jar . When using UDF jar and  creating function with this jar,  the jar 
file will not delete.
{noformat}
Beeline version 2.1.1 by Apache Hive
0: jdbc:hive2://localhost:7001/> add jar hdfs:///udf/UDF-1.0-SNAPSHOT.jar;No 
rows affected (0.45 seconds)
0: jdbc:hive2://localhost:7001/> create temporary function my_md5 as 
'com.neu.hive.UDF.Md5';No rows affected (0.082 seconds)
0: jdbc:hive2://localhost:7001/> select my_md5(name) from 
test;+---+--+
| c0 |
+---+--+
| ad6f6c09bd4accc056568714692ff4cd |
| 836db9ecc83d8397a5d0205eb8344d4c |
| 5a7fcd4f1c785c8ef4931a5a9c698ac0 |
| 0e2bb4d743f2a009d4b84a9338c98f7c |
| 3f6697692f4506cf311c95848f3536d3 |
| 0de5fc94d0ba53fc7a44f0f136e82fbb |
| 04ef3e58a6b569c1a404efb90d50f906 |
| f505e0a965c8e721eef7e111c66e5c29 |
| c053092216d6f623615d5be978b98e67 |
| 8c2c58079bc3d10be0da79c5d9940f31 |
+---+--+
10 rows selected (0.12 seconds)
{noformat}
the hiveserver2 PID is 3856，then list the file handle.
{noformat}
[hadoop@10 ~]$ lsof -p 3856 | grep SNAPSHOT.jar
java 3856 hadoop DEL REG 252,16 263184 
/data/emr/hive/tmp/2f3bc7b2-c385-4552-a54e-b575b4338d63_resources/UDF-1.0-SNAPSHOT.jar
java 3856 hadoop DEL REG 252,16 263178 
/data/emr/hive/tmp/986015db-52bf-4f6a-99f8-977de471d528_resources/UDF-4.0-SNAPSHOT.jar
java 3856 hadoop 774r REG 252,16 90885742 263178 
/data/emr/hive/tmp/986015db-52bf-4f6a-99f8-977de471d528_resources/UDF-4.0-SNAPSHOT.jar
 (deleted)
java 3856 hadoop 775r REG 252,16 104220075 263184 
/data/emr/hive/tmp/2f3bc7b2-c385-4552-a54e-b575b4338d63_resources/UDF-1.0-SNAPSHOT.jar
 (deleted)
{noformat}
 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20823) Make Compactor run in a transaction

2018-10-26 Thread Eugene Koifman (JIRA)

Eugene Koifman created HIVE-20823:
-

 Summary: Make Compactor run in a transaction
 Key: HIVE-20823
 URL: https://issues.apache.org/jira/browse/HIVE-20823
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


Have compactor open a transaction and run the job in that transaction.
# make compactor produced base/delta include this txn id in the folder name, 
e.g. base_7_c17 where 17 is the txnid.
# add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to 
record this txn id
# make sure {{AcidUtils.getAcidState()}} pays attention to this transaction on 
read and ignores this dir if this txn id is not committed in the current 
snapshot
## this means not only validWriteIdList but ValidTxnIdList should be passed 
along in config (if it isn't yet)
# once this is done, {{CompactorMR.createCompactorMarker()}} can be eliminated 
and {{AcidUtils.isValidBase}} modified accordingly
# modify Cleaner so that it doesn't clean old files until new file is visible 
to all readers
# 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20822) Improvements to push computation to JDBC from Calcite

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)

Jesus Camacho Rodriguez created HIVE-20822:
--

 Summary: Improvements to push computation to JDBC from Calcite
 Key: HIVE-20822
 URL: https://issues.apache.org/jira/browse/HIVE-20822
 Project: Hive
  Issue Type: Improvement
  Components: StorageHandler
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

2018-10-26 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/#review210118
---


Ship it!




Ship It!

- Prasanth_J


On Oct. 26, 2018, 11:23 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69148/
> ---
> 
> (Updated Oct. 26, 2018, 11:23 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> e226a1f82d44550f389308f91d578e7aa4ea170a 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
> a69046f961cdf0fff7989492c489bb62f2a66d72 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
> 4c770206fe3dcceb8570be1c1ef078b376f5cafd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 807f159daa98d40e667914adc6c53fb8ecabf998 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 012a670064c155a31dab5337dc97ac5bb7b39aec 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
> e57db935d9420508ed6091e12ca6b6cd3382db5d 
>   ql/src/test/queries/clientpositive/resourceplan.q 
> fae9701ebaeaa521904a383f5fb741c13be08d8e 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 
> c11daf728cdd5bd6fe36618aff113b3d60579129 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  90b5764a2e24d5ddf71d227f1364a0d09d63c185 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 4b7b61520a2d55635f474317053a17410f3a4bb7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  c3914b668fac18ead6196a4fc449e909f5af01b1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  47ac68c667bea8f09f5301a6364c854bc18b3c0d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  c889bbdf96b887b29be858e41ee854f0731cd5cd 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  aca5227a5bb6192da6c5f070c04d2941d636bad2 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  91ba134325094e413887a89e1d605efa99218288 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  3af2ebb253f82bb85976d229d4ac2225deffdbde 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  ee0f691b524a6e822ac14e09d24d3a49ae8565b1 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  33aa08015a9e17585c42d64d44b364be96e69eaf 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  bbb4a39ec4f6f616c9a1a9042a35cafd45cf9796 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  ea088d77fdaec85834b8fd3f01eacdfac58dd245 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>  2a2d70ae802eb6f5b9ab7f4f9519a0af30d2c5b4 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  966979891b71f1cbfe50f56c40c35af8b304c47f 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  593d562c3498660861201f58d83c27d59d184046 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

2018-10-26 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/
---

(Updated Oct. 26, 2018, 11:23 p.m.)


Review request for hive, Jason Dere and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
e226a1f82d44550f389308f91d578e7aa4ea170a 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
  metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
a69046f961cdf0fff7989492c489bb62f2a66d72 
  metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
4c770206fe3dcceb8570be1c1ef078b376f5cafd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
807f159daa98d40e667914adc6c53fb8ecabf998 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
012a670064c155a31dab5337dc97ac5bb7b39aec 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
e57db935d9420508ed6091e12ca6b6cd3382db5d 
  ql/src/test/queries/clientpositive/resourceplan.q 
fae9701ebaeaa521904a383f5fb741c13be08d8e 
  ql/src/test/results/clientpositive/llap/resourceplan.q.out 
c11daf728cdd5bd6fe36618aff113b3d60579129 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 90b5764a2e24d5ddf71d227f1364a0d09d63c185 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
4b7b61520a2d55635f474317053a17410f3a4bb7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9c158040497cd3d2762620ce35e2b46bb6d5fffe 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 c3914b668fac18ead6196a4fc449e909f5af01b1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 47ac68c667bea8f09f5301a6364c854bc18b3c0d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 c889bbdf96b887b29be858e41ee854f0731cd5cd 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 aca5227a5bb6192da6c5f070c04d2941d636bad2 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 91ba134325094e413887a89e1d605efa99218288 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 3af2ebb253f82bb85976d229d4ac2225deffdbde 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 ee0f691b524a6e822ac14e09d24d3a49ae8565b1 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 33aa08015a9e17585c42d64d44b364be96e69eaf 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 bbb4a39ec4f6f616c9a1a9042a35cafd45cf9796 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 ea088d77fdaec85834b8fd3f01eacdfac58dd245 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 2a2d70ae802eb6f5b9ab7f4f9519a0af30d2c5b4 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 966979891b71f1cbfe50f56c40c35af8b304c47f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 593d562c3498660861201f58d83c27d59d184046 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 4293579ad8b55d59f2230040f23e9a693d838ca7 


Diff: https://reviews.apache.org/r/69148/diff/3/

Changes: https://reviews.apache.org/r/69148/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

2018-10-26 Thread Sergey Shelukhin



> On Oct. 26, 2018, 7:16 p.m., Prasanth_J wrote:
> > ql/src/test/results/clientpositive/llap/resourceplan.q.out
> > Line 3918 (original), 3970 (patched)
> > 
> >
> > hmm.. expected?

This one is a bug...


> On Oct. 26, 2018, 7:16 p.m., Prasanth_J wrote:
> > ql/src/test/results/clientpositive/llap/resourceplan.q.out
> > Line 4051 (original), 4101 (patched)
> > 
> >
> > Similarly are these support to show up in clientpositive tests?

Yes, there's a setting that skips errors that Harish found, that allows one to 
avoid creating 1000 negative tests :) 
However with large out file changes like this it can be confusing to tell apart 
bugs from genuine errors.


> On Oct. 26, 2018, 7:16 p.m., Prasanth_J wrote:
> > ql/src/test/results/clientpositive/llap/resourceplan.q.out
> > Line 4713 (original), 4763 (patched)
> > 
> >
> > can you add a non-default ns test as well?

This would require a config change at minicluster init time. I added a unit 
test instead for that scenario.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/#review210105
---


On Oct. 25, 2018, 9:43 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69148/
> ---
> 
> (Updated Oct. 25, 2018, 9:43 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> e226a1f82d44550f389308f91d578e7aa4ea170a 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
> a69046f961cdf0fff7989492c489bb62f2a66d72 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
> 4c770206fe3dcceb8570be1c1ef078b376f5cafd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 807f159daa98d40e667914adc6c53fb8ecabf998 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 4de038913a5c9a2c199f71702b8f70ca84d0856b 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
> e57db935d9420508ed6091e12ca6b6cd3382db5d 
>   ql/src/test/queries/clientpositive/resourceplan.q 
> fae9701ebaeaa521904a383f5fb741c13be08d8e 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 
> c11daf728cdd5bd6fe36618aff113b3d60579129 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 4b7b61520a2d55635f474317053a17410f3a4bb7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  c3914b668fac18ead6196a4fc449e909f5af01b1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  47ac68c667bea8f09f5301a6364c854bc18b3c0d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  c889bbdf96b887b29be858e41ee854f0731cd5cd 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  aca5227a5bb6192da6c5f070c04d2941d636bad2 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  91ba134325094e413887a89e1d605efa99218288 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  3af2ebb253f82bb85976d229d4ac2225deffdbde

[jira] [Created] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)

Jesus Camacho Rodriguez created HIVE-20821:
--

 Summary: Rewrite SUM0 into SUM + COALESCE combination
 Key: HIVE-20821
 URL: https://issues.apache.org/jira/browse/HIVE-20821
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Since SUM0 is not vectorized, but SUM + COALESCE are.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20820) MV partition on clause position

2018-10-26 Thread Jesus Camacho Rodriguez (JIRA)

Jesus Camacho Rodriguez created HIVE-20820:
--

 Summary: MV partition on clause position
 Key: HIVE-20820
 URL: https://issues.apache.org/jira/browse/HIVE-20820
 Project: Hive
  Issue Type: Bug
  Components: Materialized views
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


It should obey the following syntax as per 
https://cwiki.apache.org/confluence/display/Hive/Materialized+views:
{code}
CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
  [DISABLE REWRITE]
  [COMMENT materialized_view_comment]
  [PARTITIONED ON (col_name, ...)]
  [
[ROW FORMAT row_format]
[STORED AS file_format]
  | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
  ]
  [LOCATION hdfs_path]
  [TBLPROPERTIES (property_name=property_value, ...)]
AS
;
{code}
Currently it is positioned just before TBLPROPERTIES.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20819) Leaking Metastore connections when HADOOP_USER_NAME environmental variable is set

2018-10-26 Thread Roohi Syeda (JIRA)

Roohi Syeda created HIVE-20819:
--

 Summary: Leaking Metastore connections when HADOOP_USER_NAME 
environmental variable is set
 Key: HIVE-20819
 URL: https://issues.apache.org/jira/browse/HIVE-20819
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Roohi Syeda
Assignee: Roohi Syeda


Leaking Metastore connections when HADOOP_USER_NAME environmental variable is 
set.

The connections created are in ESTABLISHED state and never closed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

2018-10-26 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/#review210105
---




ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 5451 (patched)


nit. Cannot 'create'



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 5536 (patched)


nit: Cannot create trigger



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 5550 (patched)


nit: trigger
similarly for all other exceptions below. To say what specific resource is 
being created/modified/dropped



ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java
Lines 434 (patched)


unused :)



ql/src/test/results/clientpositive/llap/resourceplan.q.out
Line 3918 (original), 3970 (patched)


hmm.. expected?



ql/src/test/results/clientpositive/llap/resourceplan.q.out
Line 4051 (original), 4101 (patched)


Similarly are these support to show up in clientpositive tests?



ql/src/test/results/clientpositive/llap/resourceplan.q.out
Line 4713 (original), 4763 (patched)


can you add a non-default ns test as well?


- Prasanth_J


On Oct. 25, 2018, 9:43 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69148/
> ---
> 
> (Updated Oct. 25, 2018, 9:43 p.m.)
> 
> 
> Review request for hive, Jason Dere and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> e226a1f82d44550f389308f91d578e7aa4ea170a 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
>   metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
> a69046f961cdf0fff7989492c489bb62f2a66d72 
>   metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
> 4c770206fe3dcceb8570be1c1ef078b376f5cafd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 807f159daa98d40e667914adc6c53fb8ecabf998 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 4de038913a5c9a2c199f71702b8f70ca84d0856b 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
> e57db935d9420508ed6091e12ca6b6cd3382db5d 
>   ql/src/test/queries/clientpositive/resourceplan.q 
> fae9701ebaeaa521904a383f5fb741c13be08d8e 
>   ql/src/test/results/clientpositive/llap/resourceplan.q.out 
> c11daf728cdd5bd6fe36618aff113b3d60579129 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 4b7b61520a2d55635f474317053a17410f3a4bb7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  c3914b668fac18ead6196a4fc449e909f5af01b1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  47ac68c667bea8f09f5301a6364c854bc18b3c0d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
>  ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  c889bbdf96b887b29be858e41ee854f0731cd5cd 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  aca5227a5bb6192da6c5f070c04d2941d636bad2 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  91ba134325094e413887a89e1d605efa99218288 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
>   
>

Re: Review Request 69107: HIVE-20512

2018-10-26 Thread Bharathkrishna Guruvayoor Murali via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69107/
---

(Updated Oct. 26, 2018, 5:13 p.m.)


Review request for hive, Antal Sinkovits, Sahil Takiar, and Vihang Karajgaonkar.


Repository: hive-git


Description
---

Improve record and memory usage logging in SparkRecordHandler


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
88dd12c05ade417aca4cdaece4448d31d4e1d65f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java
 8880bb604e088755dcfb0bcb39689702fab0cb77 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 
cb5bd7ada2d5ad4f1f654cf80ddaf4504be5d035 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java 
20e7ea0f4e8d4ff79dddeaab0406fc7350d22bd7 


Diff: https://reviews.apache.org/r/69107/diff/3/

Changes: https://reviews.apache.org/r/69107/diff/2-3/


Testing
---


Thanks,

Bharathkrishna Guruvayoor Murali

[jira] [Created] (HIVE-20818) Views created with a WHERE subquery will regard views referenced in the subquery as direct input

2018-10-26 Thread Karen Coppage (JIRA)

Karen Coppage created HIVE-20818:


 Summary: Views created with a WHERE subquery will regard views 
referenced in the subquery as direct input
 Key: HIVE-20818
 URL: https://issues.apache.org/jira/browse/HIVE-20818
 Project: Hive
  Issue Type: Bug
Reporter: Karen Coppage


If Hive is configured with an authorization hook like Sentry, and a view is 
created with a WHERE clause referencing a different view' user has no access 
to, user cannot access the view as view' is considered direct input.

For example:
{code:java}
create database db1;
create database db2;
create database db3;
 
create table db1.table1 (cola string, colb string, colc string);
insert into db1.table1 values ('a','b','c');
insert into db1.table1 values ('x','y','z');
CREATE VIEW db2.view1 AS SELECT cola, colb, colc FROM db1.table1 WHERE 
cola="x"; 
CREATE VIEW db2.view2 AS SELECT table1.cola, table1.colb, table1.colc FROM 
db1.table1 WHERE table1.cola NOT IN (SELECT view1.cola FROM db2.view1); 
create view db3.view3 as select * from db2.view2
{code}
 If test_user has read permission for only db3 (but not db1 or db2), their query
{code:java}
select * from db3.view3;{code}
will fail with :
{code:java}
Error while compiling statement: FAILED: SemanticException No valid privileges 
User test_user does not have privileges for QUERY The required privileges: 
Server=server1->Db=db2->Table=view1->action=select; {code}
WHERE IN and WHERE EXISTS cause the same issue.

Cascading views created with no WHERE clauses (i.e. with simple SELECTs and 
FROM clauses) work fine.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20817) Reading Timestamp datatype via HiveServer2 gives errors

2018-10-26 Thread mahesh kumar behera (JIRA)

mahesh kumar behera created HIVE-20817:
--

 Summary: Reading Timestamp datatype via HiveServer2 gives errors
 Key: HIVE-20817
 URL: https://issues.apache.org/jira/browse/HIVE-20817
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 4.0.0
Reporter: mahesh kumar behera
Assignee: mahesh kumar behera
 Fix For: 4.0.0


CREATE TABLE JdbcBasicRead ( empno int, desg string,empname string,doj 
timestamp,Salary float,mgrid smallint, deptno tinyint ) ROW FORMAT DELIMITED 
FIELDS TERMINATED BY ',';

LOAD DATA LOCAL INPATH '/tmp/art_jdbc/hive/input/input_7columns.txt' OVERWRITE 
INTO TABLE JdbcBasicRead;

Sample Data.
—
7369,M,SMITH,1980-12-17 17:07:29.234234,5000.00,7902,20
7499,X,ALLEN,1981-02-20 17:07:29.234234,1250.00,7698,30
7521,X,WARD,1981-02-22 17:07:29.234234,01600.57,7698,40
7566,M,JONES,1981-04-02 17:07:29.234234,02975.65,7839,10
7654,X,MARTIN,1981-09-28 17:07:29.234234,01250.00,7698,20
7698,M,BLAKE,1981-05-01 17:07:29.234234,2850.98,7839,30
7782,M,CLARK,1981-06-09 17:07:29.234234,02450.00,7839,20
—

Select statement: SELECT empno, desg, empname, doj, salary, mgrid, deptno FROM 
JdbcBasicWrite

{code}
2018-09-25T07:11:03,222 WARN [HiveServer2-Handler-Pool: Thread-83]: 
thrift.ThriftCLIService (:()) - Error fetching results:
org.apache.hive.service.cli.HiveSQLException: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.type.Timestamp cannot be cast to 
java.sql.Timestamp
at 
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:469)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:910)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
 ~[hadoop-common-3.1.1.3.0.1.0-187.jar:?]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at com.sun.proxy.$Proxy46.fetchResults(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:564) 
~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:786)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
 ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
 ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
 ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
 ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[?:1.8.0_112]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.type.Timestamp cannot be cast to 
java.sql.Timestamp
at org.apache.hive.service.cli.ColumnValue.toTColumnValue(ColumnValue.java:203) 
~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
at org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:60)

[jira] [Created] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-26 Thread Rajesh Balamohan (JIRA)

Rajesh Balamohan created HIVE-20816:
---

 Summary: FastHiveDecimal throws Exception (RuntimeException: 
Unexpected #3)
 Key: HIVE-20816
 URL: https://issues.apache.org/jira/browse/HIVE-20816
 Project: Hive
  Issue Type: Improvement
Affects Versions: 2.3.2
Reporter: Rajesh Balamohan


{noformat}
with t1 as (
...
...
)
select id, max(abs(c1))) from t1 group by id;
{noformat}

throws the following exception

{noformat}
g.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
Unexpected #3
 at 
org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)

...
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
Unexpected #3
at 
org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
at 
org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
... 18 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.RuntimeException: Unexpected #3
at 
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at 
org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
at 
org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
... 19 more

{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

2018-10-26 Thread Denys Kuzmenko via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/#review210088
---


Ship it!




Ship It!

- Denys Kuzmenko


On Oct. 26, 2018, 10:34 a.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69167/
> ---
> 
> (Updated Oct. 26, 2018, 10:34 a.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20796: jdbc URL can contain sensitive information that should not be 
> logged
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  f3b38665676391fec9b85eb9a405c14632340dc6 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  f4bdd734dc4e731dda01e6031a4115cde5571baf 
> 
> 
> Diff: https://reviews.apache.org/r/69167/diff/2/
> 
> 
> Testing
> ---
> 
> New unit test created.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

2018-10-26 Thread Laszlo Pinter via Review Board



> On Oct. 25, 2018, 4:44 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Lines 1176 (patched)
> > 
> >
> > Nit: all the trendy kids use StringBuilder now

Removed StringBuffer


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/#review210043
---


On Oct. 25, 2018, 1:36 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69167/
> ---
> 
> (Updated Oct. 25, 2018, 1:36 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20796: jdbc URL can contain sensitive information that should not be 
> logged
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  f3b38665676391fec9b85eb9a405c14632340dc6 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  f4bdd734dc4e731dda01e6031a4115cde5571baf 
> 
> 
> Diff: https://reviews.apache.org/r/69167/diff/1/
> 
> 
> Testing
> ---
> 
> New unit test created.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

2018-10-26 Thread Laszlo Pinter via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/
---

(Updated Oct. 26, 2018, 10:34 a.m.)


Review request for hive and Peter Vary.


Repository: hive-git


Description
---

HIVE-20796: jdbc URL can contain sensitive information that should not be logged


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9c158040497cd3d2762620ce35e2b46bb6d5fffe 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 f3b38665676391fec9b85eb9a405c14632340dc6 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
 f4bdd734dc4e731dda01e6031a4115cde5571baf 


Diff: https://reviews.apache.org/r/69167/diff/2/

Changes: https://reviews.apache.org/r/69167/diff/1-2/


Testing
---

New unit test created.


Thanks,

Laszlo Pinter

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

2018-10-26 Thread Laszlo Pinter via Review Board



> On Oct. 25, 2018, 4:48 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Lines 1163 (patched)
> > 
> >
> > Strings are immutable

done.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/#review210044
---


On Oct. 25, 2018, 1:36 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69167/
> ---
> 
> (Updated Oct. 25, 2018, 1:36 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20796: jdbc URL can contain sensitive information that should not be 
> logged
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  f3b38665676391fec9b85eb9a405c14632340dc6 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  f4bdd734dc4e731dda01e6031a4115cde5571baf 
> 
> 
> Diff: https://reviews.apache.org/r/69167/diff/1/
> 
> 
> Testing
> ---
> 
> New unit test created.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

2018-10-26 Thread Laszlo Pinter via Review Board



> On Oct. 25, 2018, 10:46 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Lines 1172 (patched)
> > 
> >
> > Could be optimized to:
> > 
> > String[] sensitiveData = {"user", "password"};
> > String regex = "([;,\?&]" + String.join("|", sensitiveData) + 
> > ")=.*?([;,&\)]+)";
> > 
> > String result = 
> > Pattern.compile(regex).matcher(connectionURL).replaceAll("$1=***$2");
> 
> Denys Kuzmenko wrote:
> or just connectionURL.replaceAll(regex, "$1=***$2");
> 
> Denys Kuzmenko wrote:
> regex = "([;,?&]" + String.join("|", sensitiveData) + ")=.*?([;,&)]?)";

Thanks for the input. It's much shorter than my implementation.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69167/#review210059
---


On Oct. 25, 2018, 1:36 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69167/
> ---
> 
> (Updated Oct. 25, 2018, 1:36 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20796: jdbc URL can contain sensitive information that should not be 
> logged
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9c158040497cd3d2762620ce35e2b46bb6d5fffe 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  f3b38665676391fec9b85eb9a405c14632340dc6 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
>  f4bdd734dc4e731dda01e6031a4115cde5571baf 
> 
> 
> Diff: https://reviews.apache.org/r/69167/diff/1/
> 
> 
> Testing
> ---
> 
> New unit test created.
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

[jira] [Created] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

[jira] [Created] (HIVE-20824) HiveServer2 is still leaking file handles when using ADD JAR

[jira] [Created] (HIVE-20823) Make Compactor run in a transaction

[jira] [Created] (HIVE-20822) Improvements to push computation to JDBC from Calcite

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

[jira] [Created] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination

[jira] [Created] (HIVE-20820) MV partition on clause position

[jira] [Created] (HIVE-20819) Leaking Metastore connections when HADOOP_USER_NAME environmental variable is set

Re: Review Request 69148: HIVE-20793 add RP namespacing to workload management

Re: Review Request 69107: HIVE-20512

[jira] [Created] (HIVE-20818) Views created with a WHERE subquery will regard views referenced in the subquery as direct input

[jira] [Created] (HIVE-20817) Reading Timestamp datatype via HiveServer2 gives errors

[jira] [Created] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

Re: Review Request 69167: HIVE-20796: jdbc URL can contain sensitive information that should not be logged

20 matches

Site Navigation

Mail list logo

Footer information