[jira] [Updated] (HIVE-7934) Improve column level encryption with key management
[
https://issues.apache.org/jira/browse/HIVE-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaomeng Huang updated HIVE-7934:
-
Description:
Now HIVE-6329 is a framework of column level encryption/decryption. But the
implementation in HIVE-6329 is just use Base64, it is not safe and have some
problems:
- Base64WriteOnly just be able to get the ciphertext from client for any users.
- Base64Rewriter just be able to get plaintext from client for any users.
I have an improvement based on HIVE-6329 using key management via kms.
# setup kms and set kms-acls.xml (e.g. user1 and root has permission to get key)
{code}
property
namehadoop.kms.acl.GET/name
valueuser1 root/value
description
ACL for get-key-version and get-current-key operations.
/description
/property
{code}
# create an encrypted table
{code}
-- region-aes-column.q
drop table region_aes_column;
create table region_aes_column (r_regionkey int, r_name string) ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('column.encode.columns'='r_name',
'column.encode.classname'='org.apache.hadoop.hive.serde2.aes.AESRewriter')
STORED AS TEXTFILE TBLPROPERTIES(hive.encrypt.keynames=hive.k1);
insert overwrite table region_aes_column
select
r_regionkey, r_name
from region;
{code}
# query table by different user, this is transparent to users. It is very
convenient and don't need to set anything.
{code}
[root@huang1 hive_data]# hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.9 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user1
[user1@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.899 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user2
[user2@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 RcQycWVD
1 Rc8lam9Bxg==
2 RdEpeQ==
3 Qdcyd3ZH
4 ScskfGpHp8KIIuY=
Time taken: 0.749 seconds, Fetched: 5 row(s)
{code}
was:
Now HIVE-6329 is a framework of column level encryption/decryption. But the
implementation in HIVE-6329 is just use Base64, it is not safe and have some
problems:
Base64WriteOnly can just get the ciphertext from client for any users. And
Base64Rewriter can just get plaintext from client for any users.
I have an improvement based on HIVE-6329 using key management via kms.
# setup kms and set kms-acls.xml (e.g. user1 and root has permission to get key)
{code}
property
namehadoop.kms.acl.GET/name
valueuser1 root/value
description
ACL for get-key-version and get-current-key operations.
/description
/property
{code}
# create an encrypted table
{code}
-- region-aes-column.q
drop table region_aes_column;
create table region_aes_column (r_regionkey int, r_name string) ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('column.encode.columns'='r_name',
'column.encode.classname'='org.apache.hadoop.hive.serde2.aes.AESRewriter')
STORED AS TEXTFILE TBLPROPERTIES(hive.encrypt.keynames=hive.k1);
insert overwrite table region_aes_column
select
r_regionkey, r_name
from region;
{code}
# query table by different user, this is transparent to users. It is very
convenient and don't need to set anything.
{code}
[root@huang1 hive_data]# hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.9 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user1
[user1@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.899 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user2
[user2@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 RcQycWVD
1 Rc8lam9Bxg==
2 RdEpeQ==
3 Qdcyd3ZH
4 ScskfGpHp8KIIuY=
Time taken: 0.749 seconds, Fetched: 5 row(s)
{code}
Improve column level encryption with key management
---
Key: HIVE-7934
URL: https://issues.apache.org/jira/browse/HIVE-7934
Project: Hive
Issue Type: Improvement
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Priority: Minor
Now HIVE-6329 is a framework of column level encryption/decryption. But the
implementation in HIVE-6329 is just use Base64, it is not safe and have some
problems:
- Base64WriteOnly just be able to get the ciphertext from client for any
users.
- Base64Rewriter just be able to get plaintext from client for any users.
I have an improvement based on HIVE-6329 using key management via kms.
# setup kms and set kms-acls.xml (e.g. user1 and root
[jira] [Updated] (HIVE-8042) Optionally allow move tasks to run in parallel
[ https://issues.apache.org/jira/browse/HIVE-8042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8042: - Attachment: HIVE-8042.3.patch Looked at the failures. Unrelated (tested with and without patch same result), except for the parallel.q in MR and Tez. I had updated the golden file with an extra line which tests spit out on my Mac, but that also happens with or w/o the patch. Optionally allow move tasks to run in parallel -- Key: HIVE-8042 URL: https://issues.apache.org/jira/browse/HIVE-8042 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8042.1.patch, HIVE-8042.2.patch, HIVE-8042.3.patch hive.exec.parallel allows one to run different stages of a query in parallel. However that applies only to map-reduce tasks. When using large multi insert queries there are many MoveTasks that are all executed in sequence on the client. There's no real reason for that - they could be run in parallel as well (i.e.: the stage graph captures the dependencies and knows which tasks can happen in parallel). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8042) Optionally allow move tasks to run in parallel
[ https://issues.apache.org/jira/browse/HIVE-8042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8042: - Status: Open (was: Patch Available) Optionally allow move tasks to run in parallel -- Key: HIVE-8042 URL: https://issues.apache.org/jira/browse/HIVE-8042 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8042.1.patch, HIVE-8042.2.patch, HIVE-8042.3.patch hive.exec.parallel allows one to run different stages of a query in parallel. However that applies only to map-reduce tasks. When using large multi insert queries there are many MoveTasks that are all executed in sequence on the client. There's no real reason for that - they could be run in parallel as well (i.e.: the stage graph captures the dependencies and knows which tasks can happen in parallel). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8042) Optionally allow move tasks to run in parallel
[ https://issues.apache.org/jira/browse/HIVE-8042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8042: - Status: Patch Available (was: Open) Optionally allow move tasks to run in parallel -- Key: HIVE-8042 URL: https://issues.apache.org/jira/browse/HIVE-8042 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8042.1.patch, HIVE-8042.2.patch, HIVE-8042.3.patch hive.exec.parallel allows one to run different stages of a query in parallel. However that applies only to map-reduce tasks. When using large multi insert queries there are many MoveTasks that are all executed in sequence on the client. There's no real reason for that - they could be run in parallel as well (i.e.: the stage graph captures the dependencies and knows which tasks can happen in parallel). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7934) Improve column level encryption with key management
[
https://issues.apache.org/jira/browse/HIVE-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaomeng Huang updated HIVE-7934:
-
Description:
Now HIVE-6329 is a framework of column level encryption/decryption. But the
implementation in HIVE-6329 is just use Base64, it is not safe and have some
problems:
- Base64WriteOnly just be able to get the ciphertext from client for any users.
- Base64Rewriter just be able to get plaintext from client for any users.
I have an improvement based on HIVE-6329 using key management via kms.
# setup kms and set kms-acls.xml (e.g. user1 and root has permission to get key)
{code}
property
namehadoop.kms.acl.GET/name
valueuser1 root/value
description
ACL for get-key-version and get-current-key operations.
/description
/property
{code}
# set hive-site.xml
{code}
property
namehadoop.security.kms.uri/name
valuehttp://localhost:16000/kms/value
/property
{code}
# create an encrypted table
{code}
-- region-aes-column.q
drop table region_aes_column;
create table region_aes_column (r_regionkey int, r_name string) ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('column.encode.columns'='r_name',
'column.encode.classname'='org.apache.hadoop.hive.serde2.aes.AESRewriter')
STORED AS TEXTFILE TBLPROPERTIES(hive.encrypt.keynames=hive.k1);
insert overwrite table region_aes_column
select
r_regionkey, r_name
from region;
{code}
# query table by different user, this is transparent to users. It is very
convenient and don't need to set anything.
{code}
[root@huang1 hive_data]# hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.9 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user1
[user1@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.899 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user2
[user2@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 RcQycWVD
1 Rc8lam9Bxg==
2 RdEpeQ==
3 Qdcyd3ZH
4 ScskfGpHp8KIIuY=
Time taken: 0.749 seconds, Fetched: 5 row(s)
{code}
was:
Now HIVE-6329 is a framework of column level encryption/decryption. But the
implementation in HIVE-6329 is just use Base64, it is not safe and have some
problems:
- Base64WriteOnly just be able to get the ciphertext from client for any users.
- Base64Rewriter just be able to get plaintext from client for any users.
I have an improvement based on HIVE-6329 using key management via kms.
# setup kms and set kms-acls.xml (e.g. user1 and root has permission to get key)
{code}
property
namehadoop.kms.acl.GET/name
valueuser1 root/value
description
ACL for get-key-version and get-current-key operations.
/description
/property
{code}
# create an encrypted table
{code}
-- region-aes-column.q
drop table region_aes_column;
create table region_aes_column (r_regionkey int, r_name string) ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('column.encode.columns'='r_name',
'column.encode.classname'='org.apache.hadoop.hive.serde2.aes.AESRewriter')
STORED AS TEXTFILE TBLPROPERTIES(hive.encrypt.keynames=hive.k1);
insert overwrite table region_aes_column
select
r_regionkey, r_name
from region;
{code}
# query table by different user, this is transparent to users. It is very
convenient and don't need to set anything.
{code}
[root@huang1 hive_data]# hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.9 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user1
[user1@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.899 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user2
[user2@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 RcQycWVD
1 Rc8lam9Bxg==
2 RdEpeQ==
3 Qdcyd3ZH
4 ScskfGpHp8KIIuY=
Time taken: 0.749 seconds, Fetched: 5 row(s)
{code}
Improve column level encryption with key management
---
Key: HIVE-7934
URL: https://issues.apache.org/jira/browse/HIVE-7934
Project: Hive
Issue Type: Improvement
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Priority: Minor
Now HIVE-6329 is a framework of column level encryption/decryption. But the
implementation in HIVE-6329 is just use Base64, it is not safe and have some
problems:
- Base64WriteOnly just be able to get the ciphertext from client for any
users.
- Base64Rewriter just be able to get
[jira] [Created] (HIVE-8051) Some union queries fail with dynamic partition pruning on tez
Gunther Hagleitner created HIVE-8051: Summary: Some union queries fail with dynamic partition pruning on tez Key: HIVE-8051 URL: https://issues.apache.org/jira/browse/HIVE-8051 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8051) Some union queries fail with dynamic partition pruning on tez
[ https://issues.apache.org/jira/browse/HIVE-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8051: - Description: SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' UNION ALL SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; causes AssertionError. Some union queries fail with dynamic partition pruning on tez - Key: HIVE-8051 URL: https://issues.apache.org/jira/browse/HIVE-8051 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' UNION ALL SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; causes AssertionError. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8051) Some union queries fail with dynamic partition pruning on tez
[ https://issues.apache.org/jira/browse/HIVE-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8051: - Attachment: HIVE-8051.1.patch Some union queries fail with dynamic partition pruning on tez - Key: HIVE-8051 URL: https://issues.apache.org/jira/browse/HIVE-8051 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8051.1.patch SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' UNION ALL SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; causes AssertionError. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7935) Support dynamic service discovery for HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129726#comment-14129726
]
Hive QA commented on HIVE-7935:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667863/HIVE-7935.4.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 6113 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
org.apache.hive.beeline.TestBeeLineWithArgs.org.apache.hive.beeline.TestBeeLineWithArgs
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.TestJdbcDriver2.org.apache.hive.jdbc.TestJdbcDriver2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/735/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/735/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-735/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667863
Support dynamic service discovery for HiveServer2
-
Key: HIVE-7935
URL: https://issues.apache.org/jira/browse/HIVE-7935
Project: Hive
Issue Type: New Feature
Components: HiveServer2, JDBC
Affects Versions: 0.14.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Fix For: 0.14.0
Attachments: HIVE-7935.1.patch, HIVE-7935.2.patch, HIVE-7935.3.patch,
HIVE-7935.4.patch
To support Rolling Upgrade / HA, we need a mechanism by which a JDBC client
can dynamically resolve an HiveServer2 to connect to.
*High Level Design:*
Whether, dynamic service discovery is supported or not, can be configured by
setting HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY. ZooKeeper is used to
support this.
* When an instance of HiveServer2 comes up, it adds itself as a znode to
ZooKeeper under a configurable namespace (HIVE_SERVER2_ZOOKEEPER_NAMESPACE).
* A JDBC/ODBC client now specifies the ZooKeeper ensemble in its connection
string, instead of pointing to a specific HiveServer2 instance. The JDBC
driver, uses the ZooKeeper ensemble to pick an instance of HiveServer2 to
connect for the entire session.
* When an instance is removed from ZooKeeper, the existing client sessions
continue till completion. When the last client session completes, the
instance shuts down.
* All new client connection pick one of the available HiveServer2 uris from
ZooKeeper.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8051) Some union queries fail with dynamic partition pruning on tez
[ https://issues.apache.org/jira/browse/HIVE-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8051: - Status: Patch Available (was: Open) Some union queries fail with dynamic partition pruning on tez - Key: HIVE-8051 URL: https://issues.apache.org/jira/browse/HIVE-8051 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8051.1.patch SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' UNION ALL SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; causes AssertionError. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8051) Some union queries fail with dynamic partition pruning on tez
[ https://issues.apache.org/jira/browse/HIVE-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129729#comment-14129729 ] Gunther Hagleitner commented on HIVE-8051: -- One line fix. [~vikram.dixit] can you take a look? Some union queries fail with dynamic partition pruning on tez - Key: HIVE-8051 URL: https://issues.apache.org/jira/browse/HIVE-8051 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8051.1.patch SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' UNION ALL SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; causes AssertionError. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8049) Transparent column level encryption using key management
[ https://issues.apache.org/jira/browse/HIVE-8049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129787#comment-14129787 ] Xiaomeng Huang commented on HIVE-8049: -- Initial patch based on kms simple mode Transparent column level encryption using key management Key: HIVE-8049 URL: https://issues.apache.org/jira/browse/HIVE-8049 Project: Hive Issue Type: Sub-task Reporter: Xiaomeng Huang Assignee: Xiaomeng Huang Attachments: HIVE-8049.001.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8049) Transparent column level encryption using key management
[
https://issues.apache.org/jira/browse/HIVE-8049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaomeng Huang updated HIVE-8049:
-
Description:
This patch implement transparent column level encryption. Users don't need to
set anything when they quey tables.
# setup kms and set kms-acls.xml (e.g. user1 and root has permission to get key)
{code}
property
namehadoop.kms.acl.GET/name
valueuser1 root/value
description
ACL for get-key-version and get-current-key operations.
/description
/property
{code}
# set hive-site.xml
{code}
property
namehadoop.security.kms.uri/name
valuehttp://localhost:16000/kms/value
/property
{code}
# create an encrypted table
{code}
-- region-aes-column.q
drop table region_aes_column;
create table region_aes_column (r_regionkey int, r_name string) ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('column.encode.columns'='r_name',
'column.encode.classname'='org.apache.hadoop.hive.serde2.aes.AESRewriter')
STORED AS TEXTFILE TBLPROPERTIES(hive.encrypt.keynames=hive.k1);
insert overwrite table region_aes_column
select
r_regionkey, r_name
from region;
{code}
# query table by different user, this is transparent to users. It is very
convenient and don't need to set anything.
{code}
[root@huang1 hive_data]# hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.9 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user1
[user1@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.899 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user2
[user2@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 RcQycWVD
1 Rc8lam9Bxg==
2 RdEpeQ==
3 Qdcyd3ZH
4 ScskfGpHp8KIIuY=
Time taken: 0.749 seconds, Fetched: 5 row(s)
{code}
Transparent column level encryption using key management
Key: HIVE-8049
URL: https://issues.apache.org/jira/browse/HIVE-8049
Project: Hive
Issue Type: Sub-task
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Attachments: HIVE-8049.001.patch
This patch implement transparent column level encryption. Users don't need to
set anything when they quey tables.
# setup kms and set kms-acls.xml (e.g. user1 and root has permission to get
key)
{code}
property
namehadoop.kms.acl.GET/name
valueuser1 root/value
description
ACL for get-key-version and get-current-key operations.
/description
/property
{code}
# set hive-site.xml
{code}
property
namehadoop.security.kms.uri/name
valuehttp://localhost:16000/kms/value
/property
{code}
# create an encrypted table
{code}
-- region-aes-column.q
drop table region_aes_column;
create table region_aes_column (r_regionkey int, r_name string) ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('column.encode.columns'='r_name',
'column.encode.classname'='org.apache.hadoop.hive.serde2.aes.AESRewriter')
STORED AS TEXTFILE TBLPROPERTIES(hive.encrypt.keynames=hive.k1);
insert overwrite table region_aes_column
select
r_regionkey, r_name
from region;
{code}
# query table by different user, this is transparent to users. It is very
convenient and don't need to set anything.
{code}
[root@huang1 hive_data]# hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.9 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user1
[user1@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 AFRICA
1 AMERICA
2 ASIA
3 EUROPE
4 MIDDLE EAST
Time taken: 0.899 seconds, Fetched: 5 row(s)
[root@huang1 hive_data]# su user2
[user2@huang1 hive_data]$ hive
hive select * from region_aes_column;
OK
0 RcQycWVD
1 Rc8lam9Bxg==
2 RdEpeQ==
3 Qdcyd3ZH
4 ScskfGpHp8KIIuY=
Time taken: 0.749 seconds, Fetched: 5 row(s)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8052) Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP
Matt McCline created HIVE-8052: -- Summary: Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP Key: HIVE-8052 URL: https://issues.apache.org/jira/browse/HIVE-8052 Project: Hive Issue Type: Bug Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Changes in HIVE-5760 to make explicit when timestamp and date can be vectorized as Long were accidentally to strict for min, max, count, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8052) Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-8052: --- Attachment: HIVE-8052.01.patch Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP --- Key: HIVE-8052 URL: https://issues.apache.org/jira/browse/HIVE-8052 Project: Hive Issue Type: Bug Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-8052.01.patch Changes in HIVE-5760 to make explicit when timestamp and date can be vectorized as Long were accidentally to strict for min, max, count, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8052) Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-8052: --- Status: Patch Available (was: Open) Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP --- Key: HIVE-8052 URL: https://issues.apache.org/jira/browse/HIVE-8052 Project: Hive Issue Type: Bug Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-8052.01.patch Changes in HIVE-5760 to make explicit when timestamp and date can be vectorized as Long were accidentally to strict for min, max, count, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7859) Tune zlib compression in ORC to account for the encoding strategy
[
https://issues.apache.org/jira/browse/HIVE-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129789#comment-14129789
]
Hive QA commented on HIVE-7859:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667899/HIVE-7859.3.patch
{color:green}SUCCESS:{color} +1 6195 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/736/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/736/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-736/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667899
Tune zlib compression in ORC to account for the encoding strategy
-
Key: HIVE-7859
URL: https://issues.apache.org/jira/browse/HIVE-7859
Project: Hive
Issue Type: Bug
Components: File Formats
Reporter: Gopal V
Assignee: Gopal V
Attachments: HIVE-7859.1.patch, HIVE-7859.2.patch, HIVE-7859.3.patch
Currently ORC Zlib is slow because several compression strategies ZLib uses
is already done by ORC in itself (dictionary, RLE, bit-packing).
We need to pick between Z_FILTERED, Z_HUFFMAN_ONLY, Z_RLE, Z_FIXED and
Z_DEFAULT_STRATEGY according to column stream type.
For instance an RLE_V2 stream could a use Z_FILTERED compression without
invoking the rest of the strategies.
The string streams can use Z_FIXED compression strategies and so on.
The core limitation to stick to retain compatibility with the default
decompressor, so that these are automatically backward compatible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7733) Ambiguous column reference error on query
[
https://issues.apache.org/jira/browse/HIVE-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-7733:
Attachment: HIVE-7733.3.patch.txt
Ambiguous column reference error on query
-
Key: HIVE-7733
URL: https://issues.apache.org/jira/browse/HIVE-7733
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Jason Dere
Assignee: Navis
Attachments: HIVE-7733.1.patch.txt, HIVE-7733.2.patch.txt,
HIVE-7733.3.patch.txt
{noformat}
CREATE TABLE agg1
(
col0 INT,
col1 STRING,
col2 DOUBLE
);
explain SELECT single_use_subq11.a1 AS a1,
single_use_subq11.a2 AS a2
FROM (SELECT Sum(agg1.col2) AS a1
FROM agg1
GROUP BY agg1.col0) single_use_subq12
JOIN (SELECT alias.a2 AS a0,
alias.a1 AS a1,
alias.a1 AS a2
FROM (SELECT agg1.col1 AS a0,
'42' AS a1,
agg1.col0 AS a2
FROM agg1
UNION ALL
SELECT agg1.col1 AS a0,
'41' AS a1,
agg1.col0 AS a2
FROM agg1) alias
GROUP BY alias.a2,
alias.a1) single_use_subq11
ON ( single_use_subq11.a0 = single_use_subq11.a0 );
{noformat}
Gets the following error:
FAILED: SemanticException [Error 10007]: Ambiguous column reference a2
Looks like this query had been working in 0.12 but starting failing with this
error in 0.13
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: In Progress (was: Patch Available)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.1.txt, HIVE-7892.patch.2.txt,
HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: (was: HIVE-7892.patch.1.txt)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: (was: HIVE-7892.patch.2.txt)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Attachment: HIVE-7892.1.patch
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.1.patch, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Mittal updated HIVE-7892:
Status: Patch Available (was: In Progress)
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.1.patch, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8051) Some union queries fail with dynamic partition pruning on tez
[ https://issues.apache.org/jira/browse/HIVE-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129816#comment-14129816 ] Vikram Dixit K commented on HIVE-8051: -- +1 LGTM. Some union queries fail with dynamic partition pruning on tez - Key: HIVE-8051 URL: https://issues.apache.org/jira/browse/HIVE-8051 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8051.1.patch SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' UNION ALL SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; causes AssertionError. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8053) Support custom path pattern when dynamic partitions are added in Hive
Satish Mittal created HIVE-8053: --- Summary: Support custom path pattern when dynamic partitions are added in Hive Key: HIVE-8053 URL: https://issues.apache.org/jira/browse/HIVE-8053 Project: Hive Issue Type: Improvement Reporter: Satish Mittal Assignee: Satish Mittal Currently when dynamic partitions get added in Hive, the path of dynamic partition gets created in native 'Hive style', i.e.key1=value/key2=value2/ Typical scenarios include: 1) insert - select, 2) export-import However if the table is external, user should be able to control the format of path created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8047) Lazy char/varchar are not using escape char defined in serde params
[
https://issues.apache.org/jira/browse/HIVE-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129836#comment-14129836
]
Hive QA commented on HIVE-8047:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667914/HIVE-8047.1.patch
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 6196 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_varchar_serde
org.apache.hive.jdbc.TestJdbcDriver2.testBuiltInUDFCol
org.apache.hive.jdbc.TestJdbcDriver2.testDataTypes
org.apache.hive.jdbc.TestJdbcDriver2.testDataTypes2
org.apache.hive.jdbc.TestJdbcDriver2.testExprCol
org.apache.hive.jdbc.TestJdbcDriver2.testFetchFirstNonMR
org.apache.hive.jdbc.TestJdbcDriver2.testFetchFirstQuery
org.apache.hive.jdbc.TestJdbcDriver2.testNullType
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/737/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/737/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-737/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667914
Lazy char/varchar are not using escape char defined in serde params
---
Key: HIVE-8047
URL: https://issues.apache.org/jira/browse/HIVE-8047
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-8047.1.patch
A table with char/varchar columns that specifies an escape character for
string data still has the escape characters in the char/varchar columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129843#comment-14129843 ] Satish Mittal commented on HIVE-6109: - Hi karthik, Currently this setting allows customizing path pattern when dynamic partitions are added through HCatalog APIs (HCatOutputFormat in case of MR job). It's not available when dynamic partitions get added through Hive query. I have created HIVE-8053 to track it. Support customized location for EXTERNAL tables created by Dynamic Partitioning --- Key: HIVE-6109 URL: https://issues.apache.org/jira/browse/HIVE-6109 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Satish Mittal Assignee: Satish Mittal Fix For: 0.13.0 Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, HIVE-6109.3.patch.txt, HIVE-6109.pdf Currently when dynamic partitions are created by HCatalog, the underlying directories for the partitions are created in a fixed 'Hive-style' format, i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of external table, user should be able to control the format of directories created for dynamic partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6799) HiveServer2 needs to map kerberos name to local name before proxy check
[ https://issues.apache.org/jira/browse/HIVE-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129851#comment-14129851 ] LINTE commented on HIVE-6799: - It seems that the patch has not been include in 0.13.1 release. Can you add it for the next release ? HiveServer2 needs to map kerberos name to local name before proxy check --- Key: HIVE-6799 URL: https://issues.apache.org/jira/browse/HIVE-6799 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Dilli Arumugam Assignee: Dilli Arumugam Attachments: HIVE-6799.1.patch, HIVE-6799.2.patch, HIVE-6799.patch HiveServer2 does not map kerberos name of authenticated principal to local name. Due to this, I get error like the following in HiveServer log: Failed to validate proxy privilage of knox/hdps.example.com for sam I have KINITED as knox/hdps.example@example.com I do have the following in core-site.xml property namehadoop.proxyuser.knox.groups/name valueusers/value /property property namehadoop.proxyuser.knox.hosts/name value*/value /property -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6799) HiveServer2 needs to map kerberos name to local name before proxy check
[ https://issues.apache.org/jira/browse/HIVE-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129874#comment-14129874 ] Vaibhav Gumashta commented on HIVE-6799: [~Alexandre LINTE] Thanks a lot for pointing out! This should be part of 14. HiveServer2 needs to map kerberos name to local name before proxy check --- Key: HIVE-6799 URL: https://issues.apache.org/jira/browse/HIVE-6799 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Dilli Arumugam Assignee: Dilli Arumugam Fix For: 0.14.0 Attachments: HIVE-6799.1.patch, HIVE-6799.2.patch, HIVE-6799.patch HiveServer2 does not map kerberos name of authenticated principal to local name. Due to this, I get error like the following in HiveServer log: Failed to validate proxy privilage of knox/hdps.example.com for sam I have KINITED as knox/hdps.example@example.com I do have the following in core-site.xml property namehadoop.proxyuser.knox.groups/name valueusers/value /property property namehadoop.proxyuser.knox.hosts/name value*/value /property -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6799) HiveServer2 needs to map kerberos name to local name before proxy check
[ https://issues.apache.org/jira/browse/HIVE-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6799: --- Fix Version/s: 0.14.0 HiveServer2 needs to map kerberos name to local name before proxy check --- Key: HIVE-6799 URL: https://issues.apache.org/jira/browse/HIVE-6799 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Dilli Arumugam Assignee: Dilli Arumugam Fix For: 0.14.0 Attachments: HIVE-6799.1.patch, HIVE-6799.2.patch, HIVE-6799.patch HiveServer2 does not map kerberos name of authenticated principal to local name. Due to this, I get error like the following in HiveServer log: Failed to validate proxy privilage of knox/hdps.example.com for sam I have KINITED as knox/hdps.example@example.com I do have the following in core-site.xml property namehadoop.proxyuser.knox.groups/name valueusers/value /property property namehadoop.proxyuser.knox.hosts/name value*/value /property -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6799) HiveServer2 needs to map kerberos name to local name before proxy check
[ https://issues.apache.org/jira/browse/HIVE-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6799: --- Affects Version/s: 0.13.1 HiveServer2 needs to map kerberos name to local name before proxy check --- Key: HIVE-6799 URL: https://issues.apache.org/jira/browse/HIVE-6799 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.1 Reporter: Dilli Arumugam Assignee: Dilli Arumugam Fix For: 0.14.0 Attachments: HIVE-6799.1.patch, HIVE-6799.2.patch, HIVE-6799.patch HiveServer2 does not map kerberos name of authenticated principal to local name. Due to this, I get error like the following in HiveServer log: Failed to validate proxy privilage of knox/hdps.example.com for sam I have KINITED as knox/hdps.example@example.com I do have the following in core-site.xml property namehadoop.proxyuser.knox.groups/name valueusers/value /property property namehadoop.proxyuser.knox.hosts/name value*/value /property -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Timeline for release of Hive 0.14
Hi Vikram, Can we also add: https://issues.apache.org/jira/browse/HIVE-6799 https://issues.apache.org/jira/browse/HIVE-7935 to the list. Thanks, --Vaibhav On Wed, Sep 10, 2014 at 12:18 AM, Satish Mittal satish.mit...@inmobi.com wrote: Hi, Can you please include HIVE-7892 (Thrift Set type not working with Hive) as well? It is under code review. Regards, Satish On Tue, Sep 9, 2014 at 2:10 PM, Suma Shivaprasad sumasai.shivapra...@gmail.com wrote: Please include https://issues.apache.org/jira/browse/HIVE-7694 as well. It is currently under review by Amareshwari and should be done in the next couple of days. Thanks Suma On Mon, Sep 8, 2014 at 5:44 PM, Alan Gates ga...@hortonworks.com wrote: I'll review that. I just need the time to test it against mysql, oracle, and hopefully sqlserver. But I think we can do this post branch if we need to, as it's a bug fix rather than a feature. Alan. Damien Carol dca...@blitzbs.com September 8, 2014 at 3:19 Same request for https://issues.apache.org/jira/browse/HIVE-7689 I already provided a patch, re-based it many times and I'm waiting for a review. Regards, Le 08/09/2014 12:08, amareshwarisr . a écrit : amareshwarisr . amareshw...@gmail.com September 8, 2014 at 3:08 Would like to include https://issues.apache.org/jira/browse/HIVE-2390 and https://issues.apache.org/jira/browse/HIVE-7936 . I can review and merge them. Thanks Amareshwari Vikram Dixit vik...@hortonworks.com September 5, 2014 at 17:53 Hi Folks, I am going to start consolidating the items mentioned in this list and create a wiki page to track it. I will wait till the end of next week to create the branch taking into account Ashutosh's request. Thanks Vikram. On Fri, Sep 5, 2014 at 5:39 PM, Ashutosh Chauhan hashut...@apache.org hashut...@apache.org Ashutosh Chauhan hashut...@apache.org September 5, 2014 at 17:39 Vikram, Some of us are working on stabilizing cbo branch and trying to get it merged into trunk. We feel we are close. May I request to defer cutting the branch for few more days? Folks interested in this can track our progress here : https://issues.apache.org/jira/browse/HIVE-7946 Thanks, Ashutosh On Fri, Aug 22, 2014 at 4:09 PM, Lars Francke lars.fran...@gmail.com lars.fran...@gmail.com Lars Francke lars.fran...@gmail.com August 22, 2014 at 16:09 Thank you for volunteering to do the release. I think a 0.14 release is a good idea. I have a couple of issues I'd like to get in too: * Either HIVE-7107[0] (Fix an issue in the HiveServer1 JDBC driver) or HIVE-6977[1] (Delete HiveServer1). The former needs a review the latter a patch * HIVE-6123[2] Checkstyle in Maven needs a review HIVE-7622[3] HIVE-7543[4] are waiting for any reviews or comments on my previous thread[5]. I'd still appreciate any helpers for reviews or even just comments. I'd feel very sad if I had done all that work for nothing. Hoping this thread gives me a wider audience. Both patches fix up issues that should have been caught in earlier reviews as they are almost all Checkstyle or other style violations but they make for huge patches. I could also create hundreds of small issues or stop doing these things entirely [0] https://issues.apache.org/jira/browse/HIVE-7107 https://issues.apache.org/jira/browse/HIVE-7107 [1] https://issues.apache.org/jira/browse/HIVE-6977 https://issues.apache.org/jira/browse/HIVE-6977 [2] https://issues.apache.org/jira/browse/HIVE-6123 https://issues.apache.org/jira/browse/HIVE-6123 [3] https://issues.apache.org/jira/browse/HIVE-7622 https://issues.apache.org/jira/browse/HIVE-7622 [4] https://issues.apache.org/jira/browse/HIVE-7543 https://issues.apache.org/jira/browse/HIVE-7543 On Fri, Aug 22, 2014 at 11:01 PM, John Pullokkaran -- Sent with Postbox http://www.getpostbox.com CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- _ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain
[jira] [Commented] (HIVE-8036) PTest SSH Options
[
https://issues.apache.org/jira/browse/HIVE-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129917#comment-14129917
]
Hive QA commented on HIVE-8036:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667938/HIVE-8036.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6195 tests executed
*Failed tests:*
{noformat}
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/738/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/738/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667938
PTest SSH Options
-
Key: HIVE-8036
URL: https://issues.apache.org/jira/browse/HIVE-8036
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-8036.patch
I'd like to be able to specify the following options:
{noformat}
StrictHostKeyChecking no
ConnectionAttempts 3
ServerAliveInterval 1
{noformat}
as a config param in the ptest config file as opposed to depending on them
set in the env.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8042) Optionally allow move tasks to run in parallel
[
https://issues.apache.org/jira/browse/HIVE-8042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129982#comment-14129982
]
Hive QA commented on HIVE-8042:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667951/HIVE-8042.3.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6196 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/739/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/739/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-739/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667951
Optionally allow move tasks to run in parallel
--
Key: HIVE-8042
URL: https://issues.apache.org/jira/browse/HIVE-8042
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-8042.1.patch, HIVE-8042.2.patch, HIVE-8042.3.patch
hive.exec.parallel allows one to run different stages of a query in parallel.
However that applies only to map-reduce tasks. When using large multi insert
queries there are many MoveTasks that are all executed in sequence on the
client. There's no real reason for that - they could be run in parallel as
well (i.e.: the stage graph captures the dependencies and knows which tasks
can happen in parallel).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 25245: Support dynamic service discovery for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/ --- (Updated Sept. 11, 2014, 1:08 p.m.) Review request for hive, Alan Gates, Navis Ryu, Szehon Ho, and Thejas Nair. Bugs: HIVE-7935 https://issues.apache.org/jira/browse/HIVE-7935 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7935 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5d2e6b0 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java ae128a9 jdbc/pom.xml 1ad13a7 jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbcfec7 jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 6e248d6 jdbc/src/java/org/apache/hive/jdbc/JdbcUriParseException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/Utils.java 58339bf jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 0919d2f ql/src/java/org/apache/hadoop/hive/ql/util/ZooKeeperHiveHelper.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 59294b1 service/src/java/org/apache/hive/service/cli/CLIService.java a0bc905 service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f5a8f27 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java b0bb8be service/src/java/org/apache/hive/service/cli/session/SessionManager.java 11d25cc service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java 2b80adc service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 443c371 service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java 4067106 service/src/java/org/apache/hive/service/server/HiveServer2.java 124996c service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java 66fc1fc Diff: https://reviews.apache.org/r/25245/diff/ Testing --- Manual testing. Thanks, Vaibhav Gumashta
[jira] [Updated] (HIVE-7935) Support dynamic service discovery for HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-7935: --- Attachment: HIVE-7935.5.patch [~thejas] v5 adds support for: 1. A notification from ZK to HiveServer2 when the corresponding node is deleted from ZK, which sets the de-registered flag true on the server instance. 2. Better naming of znode. 3. Closing of the last client session on a de-registered HiveServer2 instance shuts down that instance. 4. Znode ACL are now: OPEN_ACL_UNSAFE - meaning only read only access to the world. Support dynamic service discovery for HiveServer2 - Key: HIVE-7935 URL: https://issues.apache.org/jira/browse/HIVE-7935 Project: Hive Issue Type: New Feature Components: HiveServer2, JDBC Affects Versions: 0.14.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.14.0 Attachments: HIVE-7935.1.patch, HIVE-7935.2.patch, HIVE-7935.3.patch, HIVE-7935.4.patch, HIVE-7935.5.patch To support Rolling Upgrade / HA, we need a mechanism by which a JDBC client can dynamically resolve an HiveServer2 to connect to. *High Level Design:* Whether, dynamic service discovery is supported or not, can be configured by setting HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY. ZooKeeper is used to support this. * When an instance of HiveServer2 comes up, it adds itself as a znode to ZooKeeper under a configurable namespace (HIVE_SERVER2_ZOOKEEPER_NAMESPACE). * A JDBC/ODBC client now specifies the ZooKeeper ensemble in its connection string, instead of pointing to a specific HiveServer2 instance. The JDBC driver, uses the ZooKeeper ensemble to pick an instance of HiveServer2 to connect for the entire session. * When an instance is removed from ZooKeeper, the existing client sessions continue till completion. When the last client session completes, the instance shuts down. * All new client connection pick one of the available HiveServer2 uris from ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7086) TestHiveServer2.testConnection is failing on trunk
[ https://issues.apache.org/jira/browse/HIVE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130003#comment-14130003 ] Vaibhav Gumashta commented on HIVE-7086: Committed to trunk. Thanks for the review [~ashutoshc]! TestHiveServer2.testConnection is failing on trunk -- Key: HIVE-7086 URL: https://issues.apache.org/jira/browse/HIVE-7086 Project: Hive Issue Type: Test Components: HiveServer2, JDBC Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Vaibhav Gumashta Fix For: 0.14.0 Attachments: HIVE-7086.1.patch, HIVE-7086.2.patch, HIVE-7086.3.patch Able to repro locally on fresh checkout -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7086) TestHiveServer2.testConnection is failing on trunk
[ https://issues.apache.org/jira/browse/HIVE-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-7086: --- Resolution: Fixed Status: Resolved (was: Patch Available) TestHiveServer2.testConnection is failing on trunk -- Key: HIVE-7086 URL: https://issues.apache.org/jira/browse/HIVE-7086 Project: Hive Issue Type: Test Components: HiveServer2, JDBC Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Vaibhav Gumashta Fix For: 0.14.0 Attachments: HIVE-7086.1.patch, HIVE-7086.2.patch, HIVE-7086.3.patch Able to repro locally on fresh checkout -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25245: Support dynamic service discovery for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/#review53029 --- service/src/java/org/apache/hive/service/cli/CLIService.java https://reviews.apache.org/r/25245/#comment92378 I'll get rid of this import - leftover from different things I was trying. service/src/java/org/apache/hive/service/cli/CLIService.java https://reviews.apache.org/r/25245/#comment92379 Will get rid of this too. - Vaibhav Gumashta On Sept. 11, 2014, 1:08 p.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/ --- (Updated Sept. 11, 2014, 1:08 p.m.) Review request for hive, Alan Gates, Navis Ryu, Szehon Ho, and Thejas Nair. Bugs: HIVE-7935 https://issues.apache.org/jira/browse/HIVE-7935 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7935 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5d2e6b0 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java ae128a9 jdbc/pom.xml 1ad13a7 jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbcfec7 jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 6e248d6 jdbc/src/java/org/apache/hive/jdbc/JdbcUriParseException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/Utils.java 58339bf jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 0919d2f ql/src/java/org/apache/hadoop/hive/ql/util/ZooKeeperHiveHelper.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 59294b1 service/src/java/org/apache/hive/service/cli/CLIService.java a0bc905 service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f5a8f27 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java b0bb8be service/src/java/org/apache/hive/service/cli/session/SessionManager.java 11d25cc service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java 2b80adc service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 443c371 service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java 4067106 service/src/java/org/apache/hive/service/server/HiveServer2.java 124996c service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java 66fc1fc Diff: https://reviews.apache.org/r/25245/diff/ Testing --- Manual testing. Thanks, Vaibhav Gumashta
[jira] [Commented] (HIVE-8041) Hadoop-2 build is broken with JDK6
[
https://issues.apache.org/jira/browse/HIVE-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130037#comment-14130037
]
Alan Gates commented on HIVE-8041:
--
FWIW, it builds on my mac with JDK6. Which JDK are you seeing this with?
Hadoop-2 build is broken with JDK6
--
Key: HIVE-8041
URL: https://issues.apache.org/jira/browse/HIVE-8041
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-exec: Compilation failure
[ERROR]
/home/xzhang/apache/hive7/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java:[81,1]
illegal start of expression
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8036) PTest SSH Options
[
https://issues.apache.org/jira/browse/HIVE-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130044#comment-14130044
]
Xuefu Zhang commented on HIVE-8036:
---
+1, looks good to me.
PTest SSH Options
-
Key: HIVE-8036
URL: https://issues.apache.org/jira/browse/HIVE-8036
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-8036.patch
I'd like to be able to specify the following options:
{noformat}
StrictHostKeyChecking no
ConnectionAttempts 3
ServerAliveInterval 1
{noformat}
as a config param in the ptest config file as opposed to depending on them
set in the env.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8041) Hadoop-2 build is broken with JDK6
[
https://issues.apache.org/jira/browse/HIVE-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130047#comment-14130047
]
Xuefu Zhang commented on HIVE-8041:
---
I saw this with Oracle's JDK6 on Ubuntu.
Hadoop-2 build is broken with JDK6
--
Key: HIVE-8041
URL: https://issues.apache.org/jira/browse/HIVE-8041
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-exec: Compilation failure
[ERROR]
/home/xzhang/apache/hive7/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java:[81,1]
illegal start of expression
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7788) Generate plans for insert, update, and delete
[ https://issues.apache.org/jira/browse/HIVE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7788: - Status: Open (was: Patch Available) Generate plans for insert, update, and delete - Key: HIVE-7788 URL: https://issues.apache.org/jira/browse/HIVE-7788 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7788.2.patch, HIVE-7788.WIP.patch, HIVE-7788.patch Insert plans needs to be generated differently for ACID tables, plus we need to be able to generate plans in the semantic analyzer for update and delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7788) Generate plans for insert, update, and delete
[ https://issues.apache.org/jira/browse/HIVE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7788: - Attachment: HIVE-7788.3.patch New version of the patch that addresses many of Eugene's and Thejas' comments. Generate plans for insert, update, and delete - Key: HIVE-7788 URL: https://issues.apache.org/jira/browse/HIVE-7788 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7788.2.patch, HIVE-7788.3.patch, HIVE-7788.WIP.patch, HIVE-7788.patch Insert plans needs to be generated differently for ACID tables, plus we need to be able to generate plans in the semantic analyzer for update and delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25414: HIVE-7788 Generate plans for insert, update, and delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25414/ --- (Updated Sept. 11, 2014, 2:17 p.m.) Review request for hive, Ashutosh Chauhan, Eugene Koifman, Jason Dere, and Thejas Nair. Changes --- Version 3 of the patch that addresses many of Eugene's and Thejas' comments. Bugs: HIVE-7788 https://issues.apache.org/jira/browse/HIVE-7788 Repository: hive-git Description --- This patch adds plan generation as well as making modifications to some of the exec operators to make insert/value, update, and delete work. The patch is large, but about 2/3 of that are tests. Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5d2e6b0 data/conf/tez/hive-site.xml 0b3877c itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 1a84024 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java 9807497 itests/src/test/resources/testconfiguration.properties 99049ca metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java f1697bb ql/src/java/org/apache/hadoop/hive/ql/Context.java 7fcbe3c ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9953919 ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 4246d68 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java f018ca0 ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java e3bc3b1 ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 7f1d71b ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java b1c4441 ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 264052f ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 8354ad9 ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java 32d2f7a ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2b1a345 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 4acafba ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java 96a5d78 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java 5c711cf ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 5195748 ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 911ac8a ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 496f6a6 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 3e3926e ql/src/java/org/apache/hadoop/hive/ql/parse/StorageFormat.java ad91b0f ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 2dbf1c8 ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 6dce30c ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 5695f35 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 5164b16 ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 789c780 ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 63ecb8d ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java PRE-CREATION ql/src/test/queries/clientnegative/acid_overwrite.q PRE-CREATION ql/src/test/queries/clientnegative/delete_not_acid.q PRE-CREATION ql/src/test/queries/clientnegative/update_not_acid.q PRE-CREATION ql/src/test/queries/clientnegative/update_partition_col.q PRE-CREATION ql/src/test/queries/clientpositive/delete_all_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_all_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/delete_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_no_match.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_where_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/delete_whole_partition.q PRE-CREATION ql/src/test/queries/clientpositive/insert_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/insert_update_delete.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_dynamic_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_non_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_orig_table.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/insert_values_tmp_table.q PRE-CREATION ql/src/test/queries/clientpositive/update_after_multiple_inserts.q PRE-CREATION ql/src/test/queries/clientpositive/update_all_non_partitioned.q PRE-CREATION
[jira] [Updated] (HIVE-7788) Generate plans for insert, update, and delete
[ https://issues.apache.org/jira/browse/HIVE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7788: - Status: Patch Available (was: Open) Generate plans for insert, update, and delete - Key: HIVE-7788 URL: https://issues.apache.org/jira/browse/HIVE-7788 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7788.2.patch, HIVE-7788.3.patch, HIVE-7788.WIP.patch, HIVE-7788.patch Insert plans needs to be generated differently for ACID tables, plus we need to be able to generate plans in the semantic analyzer for update and delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8051) Some union queries fail with dynamic partition pruning on tez
[
https://issues.apache.org/jira/browse/HIVE-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130076#comment-14130076
]
Hive QA commented on HIVE-8051:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667958/HIVE-8051.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6195 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/740/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/740/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-740/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667958
Some union queries fail with dynamic partition pruning on tez
-
Key: HIVE-8051
URL: https://issues.apache.org/jira/browse/HIVE-8051
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-8051.1.patch
SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label =
'foo'
UNION ALL
SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label =
'bar';
causes AssertionError.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8036) PTest SSH Options
[
https://issues.apache.org/jira/browse/HIVE-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-8036:
---
Resolution: Fixed
Fix Version/s: 0.14.0
Status: Resolved (was: Patch Available)
Thank you! I committed this right away since I doubt anyone but myself and
Szehon have an interest in this patch.
PTest SSH Options
-
Key: HIVE-8036
URL: https://issues.apache.org/jira/browse/HIVE-8036
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.14.0
Attachments: HIVE-8036.patch
I'd like to be able to specify the following options:
{noformat}
StrictHostKeyChecking no
ConnectionAttempts 3
ServerAliveInterval 1
{noformat}
as a config param in the ptest config file as opposed to depending on them
set in the env.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5799) session/operation timeout for hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130136#comment-14130136
]
Brock Noland commented on HIVE-5799:
Hi [~navis],
I had a quick question but the code below. Is the case where operationTimeout
0 a normal case? If not, should we return false there?
{noformat}
+if (operationTimeout 0) {
+ // check only when it's in terminal state
+ return state.isTerminal() lastAccessTime + operationTimeout =
current;
+}
+return lastAccessTime + -operationTimeout = current;
{noformat}
session/operation timeout for hiveserver2
-
Key: HIVE-5799
URL: https://issues.apache.org/jira/browse/HIVE-5799
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Navis
Assignee: Navis
Priority: Minor
Labels: TODOC14
Fix For: 0.14.0
Attachments: HIVE-5799.1.patch.txt, HIVE-5799.10.patch.txt,
HIVE-5799.11.patch.txt, HIVE-5799.12.patch.txt, HIVE-5799.13.patch.txt,
HIVE-5799.14.patch.txt, HIVE-5799.15.patch.txt, HIVE-5799.16.patch.txt,
HIVE-5799.17.patch.txt, HIVE-5799.2.patch.txt, HIVE-5799.3.patch.txt,
HIVE-5799.4.patch.txt, HIVE-5799.5.patch.txt, HIVE-5799.6.patch.txt,
HIVE-5799.7.patch.txt, HIVE-5799.8.patch.txt, HIVE-5799.9.patch.txt
Need some timeout facility for preventing resource leakages from instable or
bad clients.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8052) Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP
[
https://issues.apache.org/jira/browse/HIVE-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130170#comment-14130170
]
Hive QA commented on HIVE-8052:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667967/HIVE-8052.01.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6195 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/741/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/741/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-741/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667967
Vectorization: min() on TimeStamp datatype fails with error Vector aggregate
not implemented: min for type: TIMESTAMP
---
Key: HIVE-8052
URL: https://issues.apache.org/jira/browse/HIVE-8052
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Attachments: HIVE-8052.01.patch
Changes in HIVE-5760 to make explicit when timestamp and date can be
vectorized as Long were accidentally to strict for min, max, count, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Review Request 25541: Handle union type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25541/ --- Review request for hive and John Pullokkaran. Bugs: HIVE-8048 https://issues.apache.org/jira/browse/HIVE-8048 Repository: hive-git Description --- Handle union type Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/RexNodeConverter.java 7a1e259 ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/TypeConverter.java 3a25186 Diff: https://reviews.apache.org/r/25541/diff/ Testing --- create_union_table.q,annotate_stats_select.q Thanks, Ashutosh Chauhan
[jira] [Updated] (HIVE-8048) [CBO] Handle union type
[ https://issues.apache.org/jira/browse/HIVE-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8048: --- Status: Patch Available (was: Open) [CBO] Handle union type --- Key: HIVE-8048 URL: https://issues.apache.org/jira/browse/HIVE-8048 Project: Hive Issue Type: Task Components: CBO Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-8048.cbo.patch Since optiq cant handle union type, disable cbo on queries involving union type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8048) [CBO] Handle union type
[ https://issues.apache.org/jira/browse/HIVE-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8048: --- Attachment: HIVE-8048.cbo.patch [CBO] Handle union type --- Key: HIVE-8048 URL: https://issues.apache.org/jira/browse/HIVE-8048 Project: Hive Issue Type: Task Components: CBO Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-8048.cbo.patch Since optiq cant handle union type, disable cbo on queries involving union type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7470) Wrong Thrift declaration for {{ShowCompactResponseElement}}
[
https://issues.apache.org/jira/browse/HIVE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-7470:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch checked in. Thanks Damien for fixing this.
Wrong Thrift declaration for {{ShowCompactResponseElement}}
---
Key: HIVE-7470
URL: https://issues.apache.org/jira/browse/HIVE-7470
Project: Hive
Issue Type: Bug
Components: Metastore, Thrift API
Affects Versions: 0.14.0
Reporter: Damien Carol
Assignee: Damien Carol
Priority: Minor
Labels: metastore, thrift
Fix For: 0.14.0
Attachments: HIVE-7470.1.patch, HIVE-7470.2.patch
Prerequiste :
1. Remote metastore
2. Activate ACID and compactions
3. Launch ALTER TABLE foo COMPACT 'bar'
4. Call {{show_compact()}} on remote metastore
This use case throws exception in Thrift stack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 24602: HIVE-7689 : Enable Postgres as METASTORE back-end
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24602/ --- (Updated sep. 11, 2014, 3:58 après-midi) Review request for hive. Changes --- Updated the last version of the patch Bugs: HIVE-7689 https://issues.apache.org/jira/browse/HIVE-7689 Repository: hive-git Description --- I maintain few patches to make Metastore works with Postgres back end in our production environment. The main goal of this JIRA is to push upstream these patches. This patch enable these features : * LOCKS on postgres metastore * COMPACTION on postgres metastore * TRANSACTION on postgres metastore * fix metastore update script for postgres Diffs (updated) - metastore/scripts/upgrade/postgres/hive-txn-schema-0.13.0.postgres.sql 2ebd3b0 metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java d3aa66f metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java df183a0 metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java f1697bb ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 264052f ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java b074ca9 ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 5e317ab ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsUtils.java 4625d27 Diff: https://reviews.apache.org/r/24602/diff/ Testing --- Using patched version in production. Enable concurrency with DbTxnManager. Thanks, Damien Carol
[jira] [Commented] (HIVE-7689) Enable Postgres as METASTORE back-end
[ https://issues.apache.org/jira/browse/HIVE-7689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130182#comment-14130182 ] Damien Carol commented on HIVE-7689: [~alangates] It's ok now. I verified failed tests and there are not related. I also updated review board. Enable Postgres as METASTORE back-end - Key: HIVE-7689 URL: https://issues.apache.org/jira/browse/HIVE-7689 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.14.0 Reporter: Damien Carol Assignee: Damien Carol Priority: Minor Labels: metastore, postgres Fix For: 0.14.0 Attachments: HIVE-7689.5.patch, HIVE-7689.6.patch, HIVE-7689.7.patch, HIVE-7689.8.patch, HIVE-7889.1.patch, HIVE-7889.2.patch, HIVE-7889.3.patch, HIVE-7889.4.patch I maintain few patches to make Metastore works with Postgres back end in our production environment. The main goal of this JIRA is to push upstream these patches. This patch enable LOCKS, COMPACTION and fix error in STATS on postgres metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6799) HiveServer2 needs to map kerberos name to local name before proxy check
[ https://issues.apache.org/jira/browse/HIVE-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130167#comment-14130167 ] LINTE commented on HIVE-6799: - I juste compile a 0.13.1 with the patch but apparently, the received Identity from knox to hiveserver2 is not forwarded to remote metastore when sasl client authentication is enable on metastore. Log on hiveserver2 : 2014-09-11 17:11:35,330 INFO [qtp599633906-22]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(188)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6 2014-09-11 17:11:35,353 INFO [qtp599633906-22]: hive.metastore (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with URI thrift://myremotemetastore.priv:9083 2014-09-11 17:11:35,360 ERROR [qtp599633906-22]: transport.TSaslTransport (TSaslTransport.java:open(296)) - SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:336) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaStoreClient.java:214) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2572) at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:434) at org.apache.hive.service.cli.thrift.ThriftCLIService.getDelegationToken(ThriftCLIService.java:283) at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:264) at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:191) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.HttpCLIServiceUGIProcessor$1.run(HttpCLIServiceUGIProcessor.java:65) at org.apache.hive.service.auth.HttpCLIServiceUGIProcessor$1.run(HttpCLIServiceUGIProcessor.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493) at org.apache.hive.service.auth.HttpCLIServiceUGIProcessor.process(HttpCLIServiceUGIProcessor.java:61) at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:99) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at
[jira] [Commented] (HIVE-7470) Wrong Thrift declaration for {{ShowCompactResponseElement}}
[
https://issues.apache.org/jira/browse/HIVE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130210#comment-14130210
]
Damien Carol commented on HIVE-7470:
[~alangates] Great !
Error with {{dfs -rmr file:///tmp/test}} is very strange. I can't believe that
unit test try to delete {{/tmp/test}} directory on local machine.
Wrong Thrift declaration for {{ShowCompactResponseElement}}
---
Key: HIVE-7470
URL: https://issues.apache.org/jira/browse/HIVE-7470
Project: Hive
Issue Type: Bug
Components: Metastore, Thrift API
Affects Versions: 0.14.0
Reporter: Damien Carol
Assignee: Damien Carol
Priority: Minor
Labels: metastore, thrift
Fix For: 0.14.0
Attachments: HIVE-7470.1.patch, HIVE-7470.2.patch
Prerequiste :
1. Remote metastore
2. Activate ACID and compactions
3. Launch ALTER TABLE foo COMPACT 'bar'
4. Call {{show_compact()}} on remote metastore
This use case throws exception in Thrift stack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 25492: HIVE-7936 - Thrift Union support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25492/ --- (Updated Sept. 11, 2014, 4:26 p.m.) Review request for hive, Amareshwari Sriramadasu and Ashutosh Chauhan. Changes --- Fixed review comments Bugs: HIVE-7936 https://issues.apache.org/jira/browse/HIVE-7936 Repository: hive-git Description --- ThriftDeserializer currently does not support UNION types Diffs (updated) - contrib/src/test/results/clientpositive/udf_example_arraymapstruct.q.out e876cdd data/files/complex.seq c27d5c09b1da881d8fd6fb2aaa1f5d169d1de3ae ql/src/test/queries/clientpositive/input_lazyserde.q 69c0d04 ql/src/test/results/clientnegative/describe_xpath1.q.out d81c96e ql/src/test/results/clientnegative/describe_xpath2.q.out 2bd0f06 ql/src/test/results/clientpositive/case_sensitivity.q.out 8684557 ql/src/test/results/clientpositive/columnarserde_create_shortcut.q.out 4805836 ql/src/test/results/clientpositive/input17.q.out 8fff21b ql/src/test/results/clientpositive/input5.q.out 7524ca7 ql/src/test/results/clientpositive/input_columnarserde.q.out 13cfb7f ql/src/test/results/clientpositive/input_dynamicserde.q.out ebcf1d8 ql/src/test/results/clientpositive/input_lazyserde.q.out 0f685f2 ql/src/test/results/clientpositive/input_testxpath.q.out 3f4b96e ql/src/test/results/clientpositive/input_testxpath2.q.out af1e999 ql/src/test/results/clientpositive/input_testxpath3.q.out b31b2f3 ql/src/test/results/clientpositive/input_testxpath4.q.out 3dca8bf ql/src/test/results/clientpositive/inputddl8.q.out fc13356 ql/src/test/results/clientpositive/join_thrift.q.out e1588c5 ql/src/test/results/clientpositive/udf_case_thrift.q.out 0fc8e84 ql/src/test/results/clientpositive/udf_coalesce.q.out 0d32476 ql/src/test/results/clientpositive/udf_isnull_isnotnull.q.out 1f600b4 ql/src/test/results/clientpositive/udf_size.q.out d7a4fa2 ql/src/test/results/clientpositive/union21.q.out 0e47ff4 ql/src/test/results/compiler/plan/case_sensitivity.q.xml c624418 ql/src/test/results/compiler/plan/input5.q.xml a881da3 ql/src/test/results/compiler/plan/input_testxpath.q.xml a3dacdf ql/src/test/results/compiler/plan/input_testxpath2.q.xml 7fc3a49 serde/if/test/complex.thrift 308b64c serde/src/gen/thrift/gen-cpp/complex_types.h 17991d4 serde/src/gen/thrift/gen-cpp/complex_types.cpp 9526d3d serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java e36a792 serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/PropValueUnion.java PRE-CREATION serde/src/gen/thrift/gen-py/complex/ttypes.py 7283e4c serde/src/gen/thrift/gen-rb/complex_types.rb 5527096 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java 9a226b3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java ee5b0d0 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftObjectInspectorUtils.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorUtils.java a18f4a7 serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestThriftObjectInspectors.java 5f692fb serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 7269cd0 Diff: https://reviews.apache.org/r/25492/diff/ Testing --- input_lazyserde.q Thanks, Suma Shivaprasad
[jira] [Commented] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130323#comment-14130323
]
Hive QA commented on HIVE-7892:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667976/HIVE-7892.1.patch
{color:green}SUCCESS:{color} +1 6196 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/743/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/743/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-743/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667976
Thrift Set type not working with Hive
-
Key: HIVE-7892
URL: https://issues.apache.org/jira/browse/HIVE-7892
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Satish Mittal
Assignee: Satish Mittal
Attachments: HIVE-7892.1.patch, HIVE-7892.patch.txt
Thrift supports List, Map and Struct complex types, which get mapped to
Array, Map and Struct complex types in Hive respectively. However thrift Set
type doesn't seem to be working.
Here is an example thrift struct:
{noformat}
namespace java sample.thrift
struct setrow {
1: required seti32 ids,
2: required string name,
}
{noformat}
A Hive table is created with ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
Describing the table shows:
{noformat}
hive describe settable;
OK
ids structfrom deserializer
namestringfrom deserializer
{noformat}
Issuing a select query on set column throws SemanticException:
{noformat}
hive select ids from settable;
FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
expected at the position 7 of 'struct' but '' is found.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8041) Hadoop-2 build is broken with JDK6
[
https://issues.apache.org/jira/browse/HIVE-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130340#comment-14130340
]
Satish Mittal commented on HIVE-8041:
-
I am seeing this with 1.6.0_26 Oracle java on ubuntu.
Hadoop-2 build is broken with JDK6
--
Key: HIVE-8041
URL: https://issues.apache.org/jira/browse/HIVE-8041
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-exec: Compilation failure
[ERROR]
/home/xzhang/apache/hive7/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java:[81,1]
illegal start of expression
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8040) Commit for HIVE-7925 breaks hadoop-1 build
[
https://issues.apache.org/jira/browse/HIVE-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130363#comment-14130363
]
Satish Mittal commented on HIVE-8040:
-
Also the default profile for pre-commit builds is hadoop-2. Building both
profiles per commit might be costly too. Should there be some build to test
hadoop-1 profile periodically?
Commit for HIVE-7925 breaks hadoop-1 build
--
Key: HIVE-8040
URL: https://issues.apache.org/jira/browse/HIVE-8040
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
on project hive-metastore: Compilation failure
[ERROR]
/home/xzhang/apache/hive7/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:[45,37]
package org.apache.commons.math3.stat does not exist
[ERROR] - [Help 1]
{code}
Missing pom file changes maybe?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8011) Fix couple of tests to work when FileSystem.access() is available
[ https://issues.apache.org/jira/browse/HIVE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-8011: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk, thanks for review Thejas. Fix couple of tests to work when FileSystem.access() is available - Key: HIVE-8011 URL: https://issues.apache.org/jira/browse/HIVE-8011 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-8011.1.patch Couple test failures when running against versions of Hadoop that support FileSystem.access(): TestStorageBasedMetastoreAuthorizationProviderWithACL: Fails with the extra tests added in HIVE-6093. The parent class does not override disallowCreateDatabase(), and it looks like this test should not override it either either. TestStorageBasedClientSideAuthorizationProvider: Should change the error message string we're checking against. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3790) UDF to introduce an OFFSET(day,month or year) for a given date or timestamp
[
https://issues.apache.org/jira/browse/HIVE-3790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130390#comment-14130390
]
ratnakar commented on HIVE-3790:
Could you someone update status on this function
UDF to introduce an OFFSET(day,month or year) for a given date or timestamp
Key: HIVE-3790
URL: https://issues.apache.org/jira/browse/HIVE-3790
Project: Hive
Issue Type: New Feature
Components: UDF
Affects Versions: 0.9.0
Reporter: Jithin John
Fix For: 0.9.1
Attachments: HIVE-3790.patch
Current releases of Hive lacks a generic function which would find the date
offset to a date / timestamp. Current releases have date_add (date) and
date_sub(date) which allows user to add or substract days only.But we could
not use year or month as a unit.
The Function DATE_OFFSET(date,offset,unit) returns the date offset value from
start_date according to the unit. Here the unit can be year , month and day.
The function could be used for date range queries and is more flexible than
the existing functions.
Functionality :-
Function Name: DATE_OFFSET(date,offset,unit)
Add a offset value to the unit part of the date/timestamp.
Returns the date in the format of -MM-dd .
Example: hive select date_offset('2009-07-29', -1 ,'MONTH' ) FROM src LIMIT 1
- 2009-06-29
Usage :-
Case : To calculate the expiry date of a item from manufacturing date
Table :- ITEM_TAB
Manufacturing_date |item id|store id|value|unit|price
2012-12-01|110001|00003|0.99|1.00|0.99
2012-12-02|110001|00008|0.99|0.00|0.00
2012-12-03|110001|00009|0.99|0.00|0.00
2012-12-04|110001|001112002|0.99|0.00|0.00
2012-12-05|110001|001112003|0.99|0.00|0.00
2012-12-06|110001|001112006|0.99|1.00|0.99
2012-12-07|110001|001112007|0.99|0.00|0.00
2012-12-08|110001|001112008|0.99|0.00|0.00
2012-12-09|110001|001112009|0.99|0.00|0.00
2012-12-10|110001|001112010|0.99|0.00|0.00
2012-12-11|110001|001113003|0.99|0.00|0.00
2012-12-12|110001|001113006|0.99|0.00|0.00
2012-12-13|110001|001113008|0.99|0.00|0.00
2012-12-14|110001|001113010|0.99|0.00|0.00
2012-12-15|110001|001114002|0.99|0.00|0.00
2012-12-16|110001|001114004|0.99|1.00|0.99
2012-12-17|110001|001114005|0.99|0.00|0.00
2012-12-18|110001|001121004|0.99|0.00|0.00
QUERY:
select man_date , date_offset(man_date ,5 ,'year') as expiry_date from
item_tab;
RESULT:
2012-12-01 2017-12-01
2012-12-02 2017-12-02
2012-12-03 2017-12-03
2012-12-04 2017-12-04
2012-12-05 2017-12-05
2012-12-06 2017-12-06
2012-12-07 2017-12-07
2012-12-08 2017-12-08
2012-12-09 2017-12-09
2012-12-10 2017-12-10
2012-12-11 2017-12-11
2012-12-12 2017-12-12
2012-12-13 2017-12-13
2012-12-14 2017-12-14
2012-12-15 2017-12-15
2012-12-16 2017-12-16
2012-12-17 2017-12-17
2012-12-18 2017-12-18
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8047) Lazy char/varchar are not using escape char defined in serde params
[ https://issues.apache.org/jira/browse/HIVE-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-8047: - Status: Open (was: Patch Available) Lazy char/varchar are not using escape char defined in serde params --- Key: HIVE-8047 URL: https://issues.apache.org/jira/browse/HIVE-8047 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-8047.1.patch A table with char/varchar columns that specifies an escape character for string data still has the escape characters in the char/varchar columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25541: Handle union type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25541/#review53064 --- Ship it! Ship It! - John Pullokkaran On Sept. 11, 2014, 3:57 p.m., Ashutosh Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25541/ --- (Updated Sept. 11, 2014, 3:57 p.m.) Review request for hive and John Pullokkaran. Bugs: HIVE-8048 https://issues.apache.org/jira/browse/HIVE-8048 Repository: hive-git Description --- Handle union type Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/RexNodeConverter.java 7a1e259 ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/TypeConverter.java 3a25186 Diff: https://reviews.apache.org/r/25541/diff/ Testing --- create_union_table.q,annotate_stats_select.q Thanks, Ashutosh Chauhan
[jira] [Updated] (HIVE-7859) Tune zlib compression in ORC to account for the encoding strategy
[ https://issues.apache.org/jira/browse/HIVE-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-7859: - Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~gopalv] for the patch. Tune zlib compression in ORC to account for the encoding strategy - Key: HIVE-7859 URL: https://issues.apache.org/jira/browse/HIVE-7859 Project: Hive Issue Type: Bug Components: File Formats Reporter: Gopal V Assignee: Gopal V Fix For: 0.14.0 Attachments: HIVE-7859.1.patch, HIVE-7859.2.patch, HIVE-7859.3.patch Currently ORC Zlib is slow because several compression strategies ZLib uses is already done by ORC in itself (dictionary, RLE, bit-packing). We need to pick between Z_FILTERED, Z_HUFFMAN_ONLY, Z_RLE, Z_FIXED and Z_DEFAULT_STRATEGY according to column stream type. For instance an RLE_V2 stream could a use Z_FILTERED compression without invoking the rest of the strategies. The string streams can use Z_FIXED compression strategies and so on. The core limitation to stick to retain compatibility with the default decompressor, so that these are automatically backward compatible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8054) Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch]
Xuefu Zhang created HIVE-8054: - Summary: Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch] Key: HIVE-8054 URL: https://issues.apache.org/jira/browse/HIVE-8054 Project: Hive Issue Type: Improvement Components: Spark Reporter: Xuefu Zhang Option hive.optimize.union.remove introduced in HIVE-3276 removes union operators from the operator graph in certain cases as an optimization reduce the number of MR jobs. While making sense in MR, this optimization is actually harmful to an execution engine such as Spark, which natives supports union without requiring additional jobs. This is because removing union operator creates disjointed operator graphs, each graph generating a job, and thus this optimization requires more jobs to run the query. Not to mention the additional complexity handling linked FS descriptors. I propose that we disable such optimization when the execution engine is Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8055) Code cleanup after HIVE-8054 [Spark Branch]
Xuefu Zhang created HIVE-8055: - Summary: Code cleanup after HIVE-8054 [Spark Branch] Key: HIVE-8055 URL: https://issues.apache.org/jira/browse/HIVE-8055 Project: Hive Issue Type: Task Components: Spark Reporter: Xuefu Zhang There is quite some code handling union removal optimization in SparkCompiler and related classes. We need to clean this up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25245: Support dynamic service discovery for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/#review53067 --- service/src/java/org/apache/hive/service/server/HiveServer2.java https://reviews.apache.org/r/25245/#comment92456 Should we use READ_ACL_UNSAFE here ? service/src/java/org/apache/hive/service/server/HiveServer2.java https://reviews.apache.org/r/25245/#comment92467 Its safer to first set the boolean and then set the watcher, to avoid potential race conditions. Otherwise, in theory you can have a case where node gets deleted, watcher sets boolean to false, and then this line sets it to true. Also, the this. for the function call seems unnecessary. We should also check if the exists call returns null (which indicates that file does not exist). call stop in that case ? service/src/java/org/apache/hive/service/server/HiveServer2.java https://reviews.apache.org/r/25245/#comment92474 I think we should check here as well, if it is OK to shut down the server. Consider the case of a rolling upgrade late in the night, when the cluster might not be very active. There might not be any remaining active connections on this server. And since it is removed from the zookeeper, no new connections would be established, and server would not come down by itself. - Thejas Nair On Sept. 11, 2014, 1:08 p.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/ --- (Updated Sept. 11, 2014, 1:08 p.m.) Review request for hive, Alan Gates, Navis Ryu, Szehon Ho, and Thejas Nair. Bugs: HIVE-7935 https://issues.apache.org/jira/browse/HIVE-7935 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7935 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5d2e6b0 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java ae128a9 jdbc/pom.xml 1ad13a7 jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbcfec7 jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 6e248d6 jdbc/src/java/org/apache/hive/jdbc/JdbcUriParseException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/Utils.java 58339bf jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 0919d2f ql/src/java/org/apache/hadoop/hive/ql/util/ZooKeeperHiveHelper.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 59294b1 service/src/java/org/apache/hive/service/cli/CLIService.java a0bc905 service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f5a8f27 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java b0bb8be service/src/java/org/apache/hive/service/cli/session/SessionManager.java 11d25cc service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java 2b80adc service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 443c371 service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java 4067106 service/src/java/org/apache/hive/service/server/HiveServer2.java 124996c service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java 66fc1fc Diff: https://reviews.apache.org/r/25245/diff/ Testing --- Manual testing. Thanks, Vaibhav Gumashta
[jira] [Commented] (HIVE-7733) Ambiguous column reference error on query
[
https://issues.apache.org/jira/browse/HIVE-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130482#comment-14130482
]
Hive QA commented on HIVE-7733:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12667972/HIVE-7733.3.patch.txt
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 6196 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ambiguous_col0
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ambiguous_col1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ambiguous_col2
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/744/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/744/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-744/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12667972
Ambiguous column reference error on query
-
Key: HIVE-7733
URL: https://issues.apache.org/jira/browse/HIVE-7733
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Jason Dere
Assignee: Navis
Attachments: HIVE-7733.1.patch.txt, HIVE-7733.2.patch.txt,
HIVE-7733.3.patch.txt
{noformat}
CREATE TABLE agg1
(
col0 INT,
col1 STRING,
col2 DOUBLE
);
explain SELECT single_use_subq11.a1 AS a1,
single_use_subq11.a2 AS a2
FROM (SELECT Sum(agg1.col2) AS a1
FROM agg1
GROUP BY agg1.col0) single_use_subq12
JOIN (SELECT alias.a2 AS a0,
alias.a1 AS a1,
alias.a1 AS a2
FROM (SELECT agg1.col1 AS a0,
'42' AS a1,
agg1.col0 AS a2
FROM agg1
UNION ALL
SELECT agg1.col1 AS a0,
'41' AS a1,
agg1.col0 AS a2
FROM agg1) alias
GROUP BY alias.a2,
alias.a1) single_use_subq11
ON ( single_use_subq11.a0 = single_use_subq11.a0 );
{noformat}
Gets the following error:
FAILED: SemanticException [Error 10007]: Ambiguous column reference a2
Looks like this query had been working in 0.12 but starting failing with this
error in 0.13
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8048) [CBO] Handle union type
[ https://issues.apache.org/jira/browse/HIVE-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8048: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to cbo branch. [CBO] Handle union type --- Key: HIVE-8048 URL: https://issues.apache.org/jira/browse/HIVE-8048 Project: Hive Issue Type: Task Components: CBO Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-8048.cbo.patch Since optiq cant handle union type, disable cbo on queries involving union type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8056) SessionState.dropSessionPaths should use FileSystem.getLocal(conf) to delete local files
Hari Sankar Sivarama Subramaniyan created HIVE-8056: --- Summary: SessionState.dropSessionPaths should use FileSystem.getLocal(conf) to delete local files Key: HIVE-8056 URL: https://issues.apache.org/jira/browse/HIVE-8056 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan SessionState.java uses the incorrect filesystem api call to remove local files. This should be changed to FileSystem.getLocal(conf). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8042) Optionally allow move tasks to run in parallel
[ https://issues.apache.org/jira/browse/HIVE-8042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8042: - Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk. Thank you [~ashutoshc] and [~xuefuz]! Optionally allow move tasks to run in parallel -- Key: HIVE-8042 URL: https://issues.apache.org/jira/browse/HIVE-8042 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.14.0 Attachments: HIVE-8042.1.patch, HIVE-8042.2.patch, HIVE-8042.3.patch hive.exec.parallel allows one to run different stages of a query in parallel. However that applies only to map-reduce tasks. When using large multi insert queries there are many MoveTasks that are all executed in sequence on the client. There's no real reason for that - they could be run in parallel as well (i.e.: the stage graph captures the dependencies and knows which tasks can happen in parallel). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-8056) SessionState.dropSessionPaths should use FileSystem.getLocal(conf) to delete local files
[ https://issues.apache.org/jira/browse/HIVE-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-8056 started by Hari Sankar Sivarama Subramaniyan. --- SessionState.dropSessionPaths should use FileSystem.getLocal(conf) to delete local files Key: HIVE-8056 URL: https://issues.apache.org/jira/browse/HIVE-8056 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan SessionState.java uses the incorrect filesystem api call to remove local files. This should be changed to FileSystem.getLocal(conf). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8056) SessionState.dropSessionPaths should use FileSystem.getLocal(conf) to delete local files
[ https://issues.apache.org/jira/browse/HIVE-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-8056: Attachment: HIVE-8056.1.patch cc-ing [~vgumashta] for review. Thanks Hari SessionState.dropSessionPaths should use FileSystem.getLocal(conf) to delete local files Key: HIVE-8056 URL: https://issues.apache.org/jira/browse/HIVE-8056 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-8056.1.patch SessionState.java uses the incorrect filesystem api call to remove local files. This should be changed to FileSystem.getLocal(conf). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24627/#review53087 --- Ship it! Ship It! - Vikram Dixit Kumaraswamy On Sept. 9, 2014, 7:32 a.m., Prasanth_J wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24627/ --- (Updated Sept. 9, 2014, 7:32 a.m.) Review request for hive and Gunther Hagleitner. Bugs: HIVE-7704 https://issues.apache.org/jira/browse/HIVE-7704 Repository: hive-git Description --- Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31aeba9 itests/src/test/resources/testconfiguration.properties 99049ca ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575 ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221 ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bbf3f6 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79 ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419 ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d ql/src/test/queries/clientpositive/orc_merge1.q ee65b98 ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 11c7578 ql/src/test/results/clientpositive/list_bucket_dml_10.q.out 8de452f ql/src/test/results/clientpositive/list_bucket_dml_4.q.out b1c060e ql/src/test/results/clientpositive/list_bucket_dml_6.q.out 3450d63 ql/src/test/results/clientpositive/list_bucket_dml_7.q.out f6a4cb5 ql/src/test/results/clientpositive/list_bucket_dml_9.q.out 796c7af ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648 ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469 ql/src/test/results/clientpositive/orc_createas1.q.out 993c853
[jira] [Commented] (HIVE-7704) Create tez task for fast file merging
[ https://issues.apache.org/jira/browse/HIVE-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130508#comment-14130508 ] Vikram Dixit K commented on HIVE-7704: -- +1 LGTM. Create tez task for fast file merging - Key: HIVE-7704 URL: https://issues.apache.org/jira/browse/HIVE-7704 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Attachments: HIVE-7704.1.patch, HIVE-7704.10.patch, HIVE-7704.2.patch, HIVE-7704.3.patch, HIVE-7704.4.patch, HIVE-7704.4.patch, HIVE-7704.5.patch, HIVE-7704.6.patch, HIVE-7704.7.patch, HIVE-7704.8.patch, HIVE-7704.9.patch Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8051) Some union queries fail with dynamic partition pruning on tez
[ https://issues.apache.org/jira/browse/HIVE-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8051: - Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~vikram.dixit]! Some union queries fail with dynamic partition pruning on tez - Key: HIVE-8051 URL: https://issues.apache.org/jira/browse/HIVE-8051 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.14.0 Attachments: HIVE-8051.1.patch SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' UNION ALL SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; causes AssertionError. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8052) Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130522#comment-14130522 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-8052: - lgtm. non-binding +1 Vectorization: min() on TimeStamp datatype fails with error Vector aggregate not implemented: min for type: TIMESTAMP --- Key: HIVE-8052 URL: https://issues.apache.org/jira/browse/HIVE-8052 Project: Hive Issue Type: Bug Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-8052.01.patch Changes in HIVE-5760 to make explicit when timestamp and date can be vectorized as Long were accidentally to strict for min, max, count, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-8018) Fix typo in config var name for dynamic partition pruning
[ https://issues.apache.org/jira/browse/HIVE-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-8018. -- Resolution: Fixed Fix Version/s: 0.14.0 Committed to trunk. Thanks [~leftylev] and [~lars_francke]! Fix typo in config var name for dynamic partition pruning - Key: HIVE-8018 URL: https://issues.apache.org/jira/browse/HIVE-8018 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Fix For: 0.14.0 Attachments: HIVE-8018.1.patch Found by [~leftylev] in HIVE-7826 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8021) support CTAS and insert ... select
[ https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8021: --- Attachment: HIVE-8021.patch patch that makes CTAS and insert work, with a simple test q file. I'd like HiveQA to run, so it's probably better to commit after CBO is merged into trunk support CTAS and insert ... select -- Key: HIVE-8021 URL: https://issues.apache.org/jira/browse/HIVE-8021 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8021.patch, HIVE-8021.preliminary.patch Need to send only the select part to CBO for now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25245: Support dynamic service discovery for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/#review53095 --- service/src/java/org/apache/hive/service/server/HiveServer2.java https://reviews.apache.org/r/25245/#comment92493 Should we do a 'mkdir -p' equivalent as done in ZookeeperTokenStore.ensurePath ? service/src/java/org/apache/hive/service/server/HiveServer2.java https://reviews.apache.org/r/25245/#comment92494 I think this is an important message about shut down, lets log it at WARN level. - Thejas Nair On Sept. 11, 2014, 1:08 p.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/ --- (Updated Sept. 11, 2014, 1:08 p.m.) Review request for hive, Alan Gates, Navis Ryu, Szehon Ho, and Thejas Nair. Bugs: HIVE-7935 https://issues.apache.org/jira/browse/HIVE-7935 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7935 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5d2e6b0 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java ae128a9 jdbc/pom.xml 1ad13a7 jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbcfec7 jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 6e248d6 jdbc/src/java/org/apache/hive/jdbc/JdbcUriParseException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/Utils.java 58339bf jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 0919d2f ql/src/java/org/apache/hadoop/hive/ql/util/ZooKeeperHiveHelper.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 59294b1 service/src/java/org/apache/hive/service/cli/CLIService.java a0bc905 service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f5a8f27 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java b0bb8be service/src/java/org/apache/hive/service/cli/session/SessionManager.java 11d25cc service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java 2b80adc service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 443c371 service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java 4067106 service/src/java/org/apache/hive/service/server/HiveServer2.java 124996c service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java 66fc1fc Diff: https://reviews.apache.org/r/25245/diff/ Testing --- Manual testing. Thanks, Vaibhav Gumashta
Review Request 25550: HIVE-8021 CBO: support CTAS and insert ... select
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25550/ --- Review request for hive, Ashutosh Chauhan and John Pullokkaran. Repository: hive-git Description --- see JIRA Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java dee7d7e ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java b5b2b60 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e4a30a2 ql/src/test/queries/clientpositive/insert0.q PRE-CREATION ql/src/test/results/clientpositive/insert0.q.out PRE-CREATION Diff: https://reviews.apache.org/r/25550/diff/ Testing --- Thanks, Sergey Shelukhin
[jira] [Updated] (HIVE-8021) CBO: support CTAS and insert ... select
[ https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8021: --- Summary: CBO: support CTAS and insert ... select (was: support CTAS and insert ... select) CBO: support CTAS and insert ... select --- Key: HIVE-8021 URL: https://issues.apache.org/jira/browse/HIVE-8021 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8021.patch, HIVE-8021.preliminary.patch Need to send only the select part to CBO for now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8021) CBO: support CTAS and insert ... select
[ https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130562#comment-14130562 ] Sergey Shelukhin commented on HIVE-8021: [~jpullokkaran] [~ashutoshc] fyi; RB at https://reviews.apache.org/r/25550/diff/# CBO: support CTAS and insert ... select --- Key: HIVE-8021 URL: https://issues.apache.org/jira/browse/HIVE-8021 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-8021.patch, HIVE-8021.preliminary.patch Need to send only the select part to CBO for now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8014) hive.in.test not set in MiniTezCli tests
[ https://issues.apache.org/jira/browse/HIVE-8014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8014: - Status: Patch Available (was: Open) hive.in.test not set in MiniTezCli tests Key: HIVE-8014 URL: https://issues.apache.org/jira/browse/HIVE-8014 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.13.1 Reporter: Alan Gates Assignee: Gunther Hagleitner Attachments: HIVE-8014.1.patch When the TestCli driver is run, the configuration value hive.in.test is set to true. However, when TestMiniTezCli driver is run, this value is not set, which results in some classes not realizing they are running in a test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8014) hive.in.test not set in MiniTezCli tests
[ https://issues.apache.org/jira/browse/HIVE-8014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8014: - Attachment: HIVE-8014.1.patch hive.in.test not set in MiniTezCli tests Key: HIVE-8014 URL: https://issues.apache.org/jira/browse/HIVE-8014 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.13.1 Reporter: Alan Gates Assignee: Gunther Hagleitner Attachments: HIVE-8014.1.patch When the TestCli driver is run, the configuration value hive.in.test is set to true. However, when TestMiniTezCli driver is run, this value is not set, which results in some classes not realizing they are running in a test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25245: Support dynamic service discovery for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/#review53097 --- service/src/java/org/apache/hive/service/server/HiveServer2.java https://reviews.apache.org/r/25245/#comment92500 I read some more about zookeeper ACL. Looks like just this READ_ACL_UNSAFE will not allow the hive user to delete the entry. It seems like we need to combine CREATOR_ALL_ACL and READ_ACL_UNSAFE and pass on to these functions. But looking at CREATOR_ALL_ACL code in zookeeper and the definition of auth scheme, it seems like it considers any authenticated user as owner, not just the current user. We need to clarify this with some zookeeper export. - Thejas Nair On Sept. 11, 2014, 1:08 p.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25245/ --- (Updated Sept. 11, 2014, 1:08 p.m.) Review request for hive, Alan Gates, Navis Ryu, Szehon Ho, and Thejas Nair. Bugs: HIVE-7935 https://issues.apache.org/jira/browse/HIVE-7935 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-7935 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5d2e6b0 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java ae128a9 jdbc/pom.xml 1ad13a7 jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java cbcfec7 jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 6e248d6 jdbc/src/java/org/apache/hive/jdbc/JdbcUriParseException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/Utils.java 58339bf jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientException.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 0919d2f ql/src/java/org/apache/hadoop/hive/ql/util/ZooKeeperHiveHelper.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java 59294b1 service/src/java/org/apache/hive/service/cli/CLIService.java a0bc905 service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f5a8f27 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java b0bb8be service/src/java/org/apache/hive/service/cli/session/SessionManager.java 11d25cc service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java 2b80adc service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 443c371 service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java 4067106 service/src/java/org/apache/hive/service/server/HiveServer2.java 124996c service/src/test/org/apache/hive/service/cli/session/TestSessionGlobalInitFile.java 66fc1fc Diff: https://reviews.apache.org/r/25245/diff/ Testing --- Manual testing. Thanks, Vaibhav Gumashta
[jira] [Created] (HIVE-8057) Preserve Record Schema For top level Query, subquery enable cbo
Laljo John Pullokkaran created HIVE-8057: Summary: Preserve Record Schema For top level Query, subquery enable cbo Key: HIVE-8057 URL: https://issues.apache.org/jira/browse/HIVE-8057 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7935) Support dynamic service discovery for HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130631#comment-14130631
]
Hive QA commented on HIVE-7935:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12668069/HIVE-7935.5.patch
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 6113 tests executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.org.apache.hive.beeline.TestBeeLineWithArgs
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.TestJdbcDriver2.org.apache.hive.jdbc.TestJdbcDriver2
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testConfOverlay
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatement
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testGetFunctions
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testGetInfo
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testOpenSession
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/745/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/745/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-745/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12668069
Support dynamic service discovery for HiveServer2
-
Key: HIVE-7935
URL: https://issues.apache.org/jira/browse/HIVE-7935
Project: Hive
Issue Type: New Feature
Components: HiveServer2, JDBC
Affects Versions: 0.14.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Fix For: 0.14.0
Attachments: HIVE-7935.1.patch, HIVE-7935.2.patch, HIVE-7935.3.patch,
HIVE-7935.4.patch, HIVE-7935.5.patch
To support Rolling Upgrade / HA, we need a mechanism by which a JDBC client
can dynamically resolve an HiveServer2 to connect to.
*High Level Design:*
Whether, dynamic service discovery is supported or not, can be configured by
setting HIVE_SERVER2_SUPPORT_DYNAMIC_SERVICE_DISCOVERY. ZooKeeper is used to
support this.
* When an instance of HiveServer2 comes up, it adds itself as a znode to
ZooKeeper under a configurable namespace (HIVE_SERVER2_ZOOKEEPER_NAMESPACE).
* A JDBC/ODBC client now specifies the ZooKeeper ensemble in its connection
string, instead of pointing to a specific HiveServer2 instance. The JDBC
driver, uses the ZooKeeper ensemble to pick an instance of HiveServer2 to
connect for the entire session.
* When an instance is removed from ZooKeeper, the existing client sessions
continue till completion. When the last client session completes, the
instance shuts down.
* All new client connection pick one of the available HiveServer2 uris from
ZooKeeper.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8058) Disable correlation opt for Tez
Gunther Hagleitner created HIVE-8058: Summary: Disable correlation opt for Tez Key: HIVE-8058 URL: https://issues.apache.org/jira/browse/HIVE-8058 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Correlation optimizer needs more thought on Tez. Stuff like nested tags are at odds with Tez (where we can expose more stages in a single DAG). For now: disable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8058) Disable correlation opt for Tez
[ https://issues.apache.org/jira/browse/HIVE-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8058: - Status: Patch Available (was: Open) Disable correlation opt for Tez --- Key: HIVE-8058 URL: https://issues.apache.org/jira/browse/HIVE-8058 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8058.1.patch Correlation optimizer needs more thought on Tez. Stuff like nested tags are at odds with Tez (where we can expose more stages in a single DAG). For now: disable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8059) MiniHS2 improvements
Robert Kanter created HIVE-8059: --- Summary: MiniHS2 improvements Key: HIVE-8059 URL: https://issues.apache.org/jira/browse/HIVE-8059 Project: Hive Issue Type: Improvement Components: HiveServer2, Testing Infrastructure Affects Versions: 0.13.1 Reporter: Robert Kanter In Oozie, we recently added a Hive 2 action that talks to Hive Server 2. With actions, we typically like to have unit tests that actually run some simple job with that action type. For the Hive 2 action, this required using the MiniHS2, but we ran into a few difficulties: # The artifact containing MiniHS2 (hive-it-unit) isn't published to the maven repo. Luckily, there were only 2 classes that we needed so copying them into Oozie for now wasn't a big deal. # Figuring out which system properties, config properties, etc needed to be set for MiniHS2 required looking at some code, the MiniHS2 pom file, and asking people. Better documentation on MiniHS2 would be good. # In Oozie, we use our own MiniHDFS and MiniMR clusters for tests. As-is, MiniHS2 can either start and use it's own Mini clusters, or it can use an existing cluster. However, to use an existing cluster, it will fork out to a shell script that we don't have. There was no way to tell MiniHS2 to use an existing cluster without forking. In the end, I had to hack the MiniHS2 constructor to allow this (see [here|https://github.com/apache/oozie/commit/e332299dc86298e5f6e457f12cc089c354d8db3f#diff-cfd2359ee992ff940b4b06871eb5da24R64]) It would be great if these could be fixed so that we can get rid of our hacks and just use MiniHS2 directly. Special thanks to [~brocknoland], [~szehon], and [~mohitsabharwal] for their help through this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8058) Disable correlation opt for Tez
[ https://issues.apache.org/jira/browse/HIVE-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8058: - Attachment: HIVE-8058.1.patch Disable correlation opt for Tez --- Key: HIVE-8058 URL: https://issues.apache.org/jira/browse/HIVE-8058 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8058.1.patch Correlation optimizer needs more thought on Tez. Stuff like nested tags are at odds with Tez (where we can expose more stages in a single DAG). For now: disable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8058) Disable correlation opt for Tez
[ https://issues.apache.org/jira/browse/HIVE-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130651#comment-14130651 ] Vikram Dixit K commented on HIVE-8058: -- Makes sense. +1 Disable correlation opt for Tez --- Key: HIVE-8058 URL: https://issues.apache.org/jira/browse/HIVE-8058 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8058.1.patch Correlation optimizer needs more thought on Tez. Stuff like nested tags are at odds with Tez (where we can expose more stages in a single DAG). For now: disable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-8054) Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Yang reassigned HIVE-8054: - Assignee: Na Yang Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch] -- Key: HIVE-8054 URL: https://issues.apache.org/jira/browse/HIVE-8054 Project: Hive Issue Type: Improvement Components: Spark Reporter: Xuefu Zhang Assignee: Na Yang Labels: Spark-M1 Option hive.optimize.union.remove introduced in HIVE-3276 removes union operators from the operator graph in certain cases as an optimization reduce the number of MR jobs. While making sense in MR, this optimization is actually harmful to an execution engine such as Spark, which natives supports union without requiring additional jobs. This is because removing union operator creates disjointed operator graphs, each graph generating a job, and thus this optimization requires more jobs to run the query. Not to mention the additional complexity handling linked FS descriptors. I propose that we disable such optimization when the execution engine is Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8060) [CBO] Handle tables with serde supplied schema
Ashutosh Chauhan created HIVE-8060: -- Summary: [CBO] Handle tables with serde supplied schema Key: HIVE-8060 URL: https://issues.apache.org/jira/browse/HIVE-8060 Project: Hive Issue Type: Bug Components: CBO Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Disable cbo in such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-7962) CBO Trunk Merge:Prevent Alter Table, drop,show Code paths from exercising CBO
[ https://issues.apache.org/jira/browse/HIVE-7962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-7962. Resolution: Cannot Reproduce as per last comment. Need to clean up other failures before attributing alter query failures to CBO CBO Trunk Merge:Prevent Alter Table, drop,show Code paths from exercising CBO - Key: HIVE-7962 URL: https://issues.apache.org/jira/browse/HIVE-7962 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8061) improve the speed of col stats update speed
Pengcheng Xiong created HIVE-8061: - Summary: improve the speed of col stats update speed Key: HIVE-8061 URL: https://issues.apache.org/jira/browse/HIVE-8061 Project: Hive Issue Type: Improvement Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Priority: Minor We worked hard towards faster update stats for columns of a partition of a table previously https://issues.apache.org/jira/browse/HIVE-7736 and https://issues.apache.org/jira/browse/HIVE-7876 Although there is some improvement, it is only correct in the first run. There will be duplicate column stats later. Thanks to Eugene Koifman 's comments. We fixed this in https://issues.apache.org/jira/browse/HIVE-7944 by reversing the patch. This JIRA ticket is my another try to improve the speed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
