[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389759#comment-17389759 ] Oleksiy Sayankin commented on HIVE-6679: [~kgyrtkirk] Could you please review the PR? > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Prasad Suresh Mujumdar >Assignee: Oleksiy Sayankin >Priority: Major > Labels: TODOC1.0, TODOC15, pull-request-available > Fix For: 1.3.0 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch, HIVE-6679.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386528#comment-17386528 ] Oleksiy Sayankin commented on HIVE-6679: Guys, please review the PR. > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Prasad Suresh Mujumdar >Assignee: Oleksiy Sayankin >Priority: Major > Labels: TODOC1.0, TODOC15, pull-request-available > Fix For: 1.3.0 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch, HIVE-6679.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378010#comment-17378010 ] Oleksiy Sayankin commented on HIVE-6679: Created PR with fix https://github.com/apache/hive/pull/2461 Please review > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Prasad Suresh Mujumdar >Assignee: Oleksiy Sayankin >Priority: Major > Labels: TODOC1.0, TODOC15, pull-request-available > Fix For: 1.3.0 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch, HIVE-6679.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-6679: -- Assignee: Oleksiy Sayankin (was: Navis Ryu) > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Prasad Suresh Mujumdar >Assignee: Oleksiy Sayankin >Priority: Major > Labels: TODOC1.0, TODOC15 > Fix For: 1.3.0 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch, HIVE-6679.6.patch > > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21075: Status: Patch Available (was: In Progress) > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21075.2.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21075: Attachment: HIVE-21075.2.patch > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21075.2.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21075 started by Oleksiy Sayankin. --- > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[
https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351720#comment-17351720
]
Oleksiy Sayankin edited comment on HIVE-21075 at 5/26/21, 11:23 AM:
Created [PR|https://github.com/apache/hive/pull/2323].
Solution is simple:
{code}
if (isPostgres()){
// use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1
} else {
// use select count(1) from
org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd ==
inCD)
}
{code}
was (Author: osayankin):
Created [PR|https://github.com/apache/hive/pull/2323].
Solution is simple:
{code}
if (isPostgres()){
// use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1
} else
{
// use select count(1) from
org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd ==
inCD)
}
{code}
> Metastore: Drop partition performance downgrade with Postgres DB
>
>
> Key: HIVE-21075
> URL: https://issues.apache.org/jira/browse/HIVE-21075
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Yongzhi Chen
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In order to workaround oracle not supporting limit statement caused
> performance issue, HIVE-9447 makes all the backend DB run select count(1)
> from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in
> SDS table before drop a partition. This select count(1) statement does not
> scale well in Postgres, and there is no index for CD_ID column in SDS table.
> For a SDS table with with 1.5 million rows, select count(1) has average 700ms
> without index, while in 10-20ms with index. But the statement before
> HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses
> less than 10ms .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[
https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351720#comment-17351720
]
Oleksiy Sayankin edited comment on HIVE-21075 at 5/26/21, 11:22 AM:
Created [PR|https://github.com/apache/hive/pull/2323].
Solution is simple:
{code}
if (isPostgres()){
// use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1
} else
{
// use select count(1) from
org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd ==
inCD)
}
{code}
was (Author: osayankin):
Created [PR|https://github.com/apache/hive/pull/2323].
Solution is simple:
{code}
if (isPostgres()){
// use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1
} else
{
// select count(1) from
org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd ==
inCD)
}
{code}
> Metastore: Drop partition performance downgrade with Postgres DB
>
>
> Key: HIVE-21075
> URL: https://issues.apache.org/jira/browse/HIVE-21075
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Yongzhi Chen
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In order to workaround oracle not supporting limit statement caused
> performance issue, HIVE-9447 makes all the backend DB run select count(1)
> from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in
> SDS table before drop a partition. This select count(1) statement does not
> scale well in Postgres, and there is no index for CD_ID column in SDS table.
> For a SDS table with with 1.5 million rows, select count(1) has average 700ms
> without index, while in 10-20ms with index. But the statement before
> HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses
> less than 10ms .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[
https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351720#comment-17351720
]
Oleksiy Sayankin commented on HIVE-21075:
-
Created [PR|https://github.com/apache/hive/pull/2323].
Solution is simple:
{code}
if (isPostgres()){
// use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1
} else
{
// select count(1) from
org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd ==
inCD)
}
{code}
> Metastore: Drop partition performance downgrade with Postgres DB
>
>
> Key: HIVE-21075
> URL: https://issues.apache.org/jira/browse/HIVE-21075
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Yongzhi Chen
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In order to workaround oracle not supporting limit statement caused
> performance issue, HIVE-9447 makes all the backend DB run select count(1)
> from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in
> SDS table before drop a partition. This select count(1) statement does not
> scale well in Postgres, and there is no index for CD_ID column in SDS table.
> For a SDS table with with 1.5 million rows, select count(1) has average 700ms
> without index, while in 10-20ms with index. But the statement before
> HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses
> less than 10ms .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-21075: --- Assignee: Oleksiy Sayankin > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[
https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-24904:
---
Assignee: Zoltan Haindrich
> CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
> --
>
> Key: HIVE-24904
> URL: https://issues.apache.org/jira/browse/HIVE-24904
> Project: Hive
> Issue Type: Bug
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Zoltan Haindrich
>Priority: Critical
> Labels: CVE
>
> CVE list: CVE-2019-10172,CVE-2019-10202
> CVSS score: High
> {code}
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[
https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-24904:
Labels: CVE (was: )
> CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
> --
>
> Key: HIVE-24904
> URL: https://issues.apache.org/jira/browse/HIVE-24904
> Project: Hive
> Issue Type: Bug
> Components: Security
>Reporter: Oleksiy Sayankin
>Priority: Critical
> Labels: CVE
>
> CVE list: CVE-2019-10172,CVE-2019-10202
> CVSS score: High
> {code}
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[
https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304055#comment-17304055
]
Oleksiy Sayankin edited comment on HIVE-24904 at 3/18/21, 11:23 AM:
The latest supported release of the lib is 1.9.13
([https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl])
for updating the lib to version with fix we have 3 options:
1.
[https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl/1.9.14.jdk17-redhat-1]
update to lib that was bundled by RedHat
2. Build our own lib from the master: [https://github.com/FasterXML/jackson-1]
3. Move to new artifact
{panel}
com.fasterxml.jackson.core » jackson-databind
{panel}
FYI: [~kgyrtkirk], [~jcamachorodriguez], [~pvary]
was (Author: osayankin):
The latest supported release of the lib is 1.9.13
([https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl])
for updating the lib to version with fix we have 3 options:
1.
[https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl/1.9.14.jdk17-redhat-1]
update to lib that was bundled by RedHat
2. Build our own lib from the master: [https://github.com/FasterXML/jackson-1]
3. Move to new artifact
{panel}
com.fasterxml.jackson.core » jackson-databind{panel}
> CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
> --
>
> Key: HIVE-24904
> URL: https://issues.apache.org/jira/browse/HIVE-24904
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Priority: Critical
>
> CVE list: CVE-2019-10172,CVE-2019-10202
> CVSS score: High
> {code}
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[
https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-24904:
Component/s: Security
> CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
> --
>
> Key: HIVE-24904
> URL: https://issues.apache.org/jira/browse/HIVE-24904
> Project: Hive
> Issue Type: Bug
> Components: Security
>Reporter: Oleksiy Sayankin
>Priority: Critical
>
> CVE list: CVE-2019-10172,CVE-2019-10202
> CVSS score: High
> {code}
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[
https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304055#comment-17304055
]
Oleksiy Sayankin commented on HIVE-24904:
-
The latest supported release of the lib is 1.9.13
([https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl])
for updating the lib to version with fix we have 3 options:
1.
[https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl/1.9.14.jdk17-redhat-1]
update to lib that was bundled by RedHat
2. Build our own lib from the master: [https://github.com/FasterXML/jackson-1]
3. Move to new artifact
{panel}
com.fasterxml.jackson.core » jackson-databind{panel}
> CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
> --
>
> Key: HIVE-24904
> URL: https://issues.apache.org/jira/browse/HIVE-24904
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Priority: Critical
>
> CVE list: CVE-2019-10172,CVE-2019-10202
> CVSS score: High
> {code}
> ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-24740:
Description:
{code}
CREATE TABLE t1 (column1 STRING);
{code}
{code}
select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4)
order by column1;
{code}
{code}
org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table
alias or column reference 'column1': (possible column names are: _c0, _c1,
.(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg
(tok_table_or_col column1)))
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810)
at
org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
[jira] [Updated] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-24740:
Description:
{code}
CREATE TABLE t1 (column1 STRING);
{code}
{code}
select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4)
order by column1;
{code}
{code}
org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table
alias or column reference 'column1': (possible column names are: _c0, _c1,
.(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg
(tok_table_or_col column1)))
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810)
at
org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
[jira] [Commented] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279025#comment-17279025
]
Oleksiy Sayankin commented on HIVE-24740:
-
[~pxiong] could you please take a look here?
> Invalid table alias or column reference: Can't order by an unselected column
>
>
> Key: HIVE-24740
> URL: https://issues.apache.org/jira/browse/HIVE-24740
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Pengcheng Xiong
>Priority: Blocker
>
> {code}
> CREATE TABLE t1 (column1 STRING);
> {code}
> {code}
> select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4)
> order by column1;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table
> alias or column reference 'column1': (possible column names are: _c0, _c1,
> .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg
> (tok_table_or_col column1)))
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
>
[jira] [Assigned] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-24740:
---
Assignee: Pengcheng Xiong (was: Peng Cheng)
> Invalid table alias or column reference: Can't order by an unselected column
>
>
> Key: HIVE-24740
> URL: https://issues.apache.org/jira/browse/HIVE-24740
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Pengcheng Xiong
>Priority: Blocker
>
> {code}
> CREATE TABLE t1 (column1 STRING);
> {code}
> {code}
> select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4)
> order by column1;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table
> alias or column reference 'column1': (possible column names are: _c0, _c1,
> .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg
> (tok_table_or_col column1)))
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
>
[jira] [Assigned] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-24740:
---
Assignee: Peng Cheng
> Invalid table alias or column reference: Can't order by an unselected column
>
>
> Key: HIVE-24740
> URL: https://issues.apache.org/jira/browse/HIVE-24740
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Peng Cheng
>Priority: Blocker
>
> {code}
> CREATE TABLE t1 (column1 STRING);
> {code}
> {code}
> select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4)
> order by column1;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table
> alias or column reference 'column1': (possible column names are: _c0, _c1,
> .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg
> (tok_table_or_col column1)))
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
>
[jira] [Updated] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-24740:
Summary: Invalid table alias or column reference: Can't order by an
unselected column (was: Can't order by an unselected column)
> Invalid table alias or column reference: Can't order by an unselected column
>
>
> Key: HIVE-24740
> URL: https://issues.apache.org/jira/browse/HIVE-24740
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Priority: Blocker
>
> {code}
> CREATE TABLE t1 (column1 STRING);
> {code}
> {code}
> select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4)
> order by column1;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table
> alias or column reference 'column1': (possible column names are: _c0, _c1,
> .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg
> (tok_table_or_col column1)))
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>
[jira] [Updated] (HIVE-24740) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-24740:
Priority: Blocker (was: Major)
> Can't order by an unselected column
> ---
>
> Key: HIVE-24740
> URL: https://issues.apache.org/jira/browse/HIVE-24740
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Priority: Blocker
>
> {code}
> CREATE TABLE t1 (column1 STRING);
> {code}
> {code}
> select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4)
> order by column1;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table
> alias or column reference 'column1': (possible column names are: _c0, _c1,
> .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg
> (tok_table_or_col column1)))
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
>
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[
https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-15444:
Status: Patch Available (was: In Progress)
> tez.queue.name is invalid after tez job running on CLI
> --
>
> Key: HIVE-15444
> URL: https://issues.apache.org/jira/browse/HIVE-15444
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.1.1
>Reporter: Hui Fei
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-15444.1.patch, HIVE-15444.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set tez.queue.name=HQ_OLPS;
> hive> set tez.queue.name;
> tez.queue.name=HQ_OLPS
> {code}
> {code}
> hive> insert into abc values(2,2);
> Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1481877998678_0011)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. container SUCCEEDED 1 100
> 0 0
> --
> VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s
> --
> Loading data to table default.abc
> OK
> Time taken: 19.983 seconds
> {code}
> {code}
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[
https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254765#comment-17254765
]
Oleksiy Sayankin commented on HIVE-15444:
-
These are results of verification of
[HIVE-14397|https://issues.apache.org/jira/browse/HIVE-14397] with disabled
queue name removal via {{hive.server2.tez.unset.tez.queue.name = false}}
{code}
conf.unset(TezConfiguration.TEZ_QUEUE_NAME)
{code}
statement.
*1.*
Configure Hive on Tez.
*2.*
Configure in {{hive-site.xml}}
{code}
hive.server2.tez.unset.tez.queue.name
false
hive.server2.tez.default.queues
q1,q2
{code}
Here we define two queues for Tez jobs: q1 and q2. We also disable removing Tez
queue name after query execution.
*3.*
Configure in {{tez-site.xml}}
{code}
tez.session.am.dag.submit.timeout.secs
60
{code}
This property allows to set Tez session timeout to 60 seconds.
*4.*
In console 1: run Beeline, connect to HS2 and set
{code}
0: jdbc:hive2://node1.cluster.com:1/defau> set tez.queue.name=q1;
{code}
*5.*
In console 2: run Beeline, connect to HS2 and set
{code}
0: jdbc:hive2://node1.cluster.com:1/defau> set tez.queue.name=q2;
{code}
*6.*
In console 1: run any simple query that triggers Tez job
{code}
0: jdbc:hive2://node1.cluster.com:1/defau> select sum(id) from test;
{code}
*7.*
In console 2: run any simple query that triggers Tez job
{code}
0: jdbc:hive2://node1.cluster.com:1/defau> select sum(id) from test;
{code}
*8.*
See two YARN applications: each for a queue: q1 and q2.
{code}
[myuser@node1 ~]$ yarn application -list
{code}
{code}
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):2
Application-Id Application-NameApplication-Type
User Queue State Final-State
ProgressTracking-URL
application_1608718387253_0011 HIVE-e5b114cc-5b31-465e-9204-3b49daf71038
TEZ myuser root.q2 RUNNING
UNDEFINED 0% http://node1.cluster.com:46621/ui/
application_1608718387253_0012 HIVE-b307590b-1fe6-445c-bf6b-57e72d88d92d
TEZ myuser root.q1 RUNNING
UNDEFINED 0% http://node1.cluster.com:42410/ui/
{code}
*9.*
Wait for 60 seconds ans see that Tez sessions are expired
{code}
[myuser@node1 ~]$ yarn application -list
{code}
{code}
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):0
Application-Id Application-NameApplication-Type
User Queue State Final-State
ProgressTracking-URL
[myuser@node1 ~]$
{code}
*10.*
Run queries from console 1 and 2 and check that there is no 4 YARN applications
after that:
{code}
[myuser@node1 ~]$ yarn application -list
{code}
{code}
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):2
Application-Id Application-NameApplication-Type
User Queue State Final-State
ProgressTracking-URL
application_1608718387253_0011 HIVE-e5b114cc-5b31-465e-9204-3b49daf71038
TEZ myuser root.q2 RUNNING
UNDEFINED 0% http://node1.cluster.com:46621/ui/
application_1608718387253_0012 HIVE-b307590b-1fe6-445c-bf6b-57e72d88d92d
TEZ myuser root.q1 RUNNING
UNDEFINED 0% http://node1.cluster.com:42410/ui/
{code}
*RESOLUTION*
We did not observe
[HIVE-14397|https://issues.apache.org/jira/browse/HIVE-14397] after applying
the fix.
> tez.queue.name is invalid after tez job running on CLI
> --
>
> Key: HIVE-15444
> URL: https://issues.apache.org/jira/browse/HIVE-15444
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Hui Fei
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-15444.1.patch, HIVE-15444.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set tez.queue.name=HQ_OLPS;
> hive> set tez.queue.name;
> tez.queue.name=HQ_OLPS
> {code}
> {code}
> hive> insert into abc values(2,2);
> Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1
> Total jobs = 1
> Launching Job 1 out of 1
>
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[
https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-15444:
Attachment: HIVE-15444.2.patch
> tez.queue.name is invalid after tez job running on CLI
> --
>
> Key: HIVE-15444
> URL: https://issues.apache.org/jira/browse/HIVE-15444
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Hui Fei
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-15444.1.patch, HIVE-15444.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set tez.queue.name=HQ_OLPS;
> hive> set tez.queue.name;
> tez.queue.name=HQ_OLPS
> {code}
> {code}
> hive> insert into abc values(2,2);
> Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1481877998678_0011)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. container SUCCEEDED 1 100
> 0 0
> --
> VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s
> --
> Loading data to table default.abc
> OK
> Time taken: 19.983 seconds
> {code}
> {code}
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[
https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-15444:
Description:
{code}
hive> set tez.queue.name;
tez.queue.name is undefined
hive> set tez.queue.name=HQ_OLPS;
hive> set tez.queue.name;
tez.queue.name=HQ_OLPS
{code}
{code}
hive> insert into abc values(2,2);
Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id
application_1481877998678_0011)
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. container SUCCEEDED 1 100
0 0
--
VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s
--
Loading data to table default.abc
OK
Time taken: 19.983 seconds
{code}
{code}
hive> set tez.queue.name;
tez.queue.name is undefined
hive> set hive.execution.engine;
hive.execution.engine=tez
{code}
was:
hive> set tez.queue.name;
tez.queue.name is undefined
hive> set tez.queue.name=HQ_OLPS;
hive> set tez.queue.name;
tez.queue.name=HQ_OLPS
hive> insert into abc values(2,2);
Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id
application_1481877998678_0011)
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. container SUCCEEDED 1 100
0 0
--
VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s
--
Loading data to table default.abc
OK
Time taken: 19.983 seconds
hive> set tez.queue.name;
tez.queue.name is undefined
hive> set hive.execution.engine;
hive.execution.engine=tez
> tez.queue.name is invalid after tez job running on CLI
> --
>
> Key: HIVE-15444
> URL: https://issues.apache.org/jira/browse/HIVE-15444
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Hui Fei
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-15444.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set tez.queue.name=HQ_OLPS;
> hive> set tez.queue.name;
> tez.queue.name=HQ_OLPS
> {code}
> {code}
> hive> insert into abc values(2,2);
> Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1481877998678_0011)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. container SUCCEEDED 1 100
> 0 0
> --
> VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s
> --
> Loading data to table default.abc
> OK
> Time taken: 19.983 seconds
> {code}
> {code}
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254762#comment-17254762 ] Oleksiy Sayankin commented on HIVE-15444: - Created https://github.com/apache/hive/pull/1815 > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 2.2.0 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-15444: --- Assignee: Oleksiy Sayankin (was: Hui Fei) > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 2.2.0 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch > > > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-15444: Status: In Progress (was: Patch Available) > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0, 2.1.1 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch > > > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21489) EXPLAIN command throws ClassCastException in Hive
[ https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247291#comment-17247291 ] Oleksiy Sayankin commented on HIVE-21489: - I've tested the patched and it works for Storage Base Authorization as well. > EXPLAIN command throws ClassCastException in Hive > - > > Key: HIVE-21489 > URL: https://issues.apache.org/jira/browse/HIVE-21489 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Ping Lu >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch > > > I'm trying to run commands like explain select * from src in hive-2.3.4,but > it falls with the ClassCastException: > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer > Steps to reproduce: > 1）hive.execution.engine is the default value mr > 2）hive.security.authorization.enabled is set to true, and > hive.security.authorization.manager is set to > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider > 3）start hivecli to run command：explain select * from src > I debug the code and find the issue HIVE-18778 causing the above > ClassCastException.If I set hive.in.test to true，the explain command can be > successfully executed。 > Now,I have one question,due to hive.in.test cann't be modified at runtime.how > to run explain command with using default authorization in hive-2.3.4, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21489) EXPLAIN command throw ClassCastException in Hive
[ https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21489: Summary: EXPLAIN command throw ClassCastException in Hive (was: explain command throw ClassCastException in 2.3.4) > EXPLAIN command throw ClassCastException in Hive > > > Key: HIVE-21489 > URL: https://issues.apache.org/jira/browse/HIVE-21489 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Ping Lu >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch > > > I'm trying to run commands like explain select * from src in hive-2.3.4,but > it falls with the ClassCastException: > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer > Steps to reproduce: > 1）hive.execution.engine is the default value mr > 2）hive.security.authorization.enabled is set to true, and > hive.security.authorization.manager is set to > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider > 3）start hivecli to run command：explain select * from src > I debug the code and find the issue HIVE-18778 causing the above > ClassCastException.If I set hive.in.test to true，the explain command can be > successfully executed。 > Now,I have one question,due to hive.in.test cann't be modified at runtime.how > to run explain command with using default authorization in hive-2.3.4, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21489) EXPLAIN command throws ClassCastException in Hive
[ https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21489: Summary: EXPLAIN command throws ClassCastException in Hive (was: EXPLAIN command throw ClassCastException in Hive) > EXPLAIN command throws ClassCastException in Hive > - > > Key: HIVE-21489 > URL: https://issues.apache.org/jira/browse/HIVE-21489 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Ping Lu >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch > > > I'm trying to run commands like explain select * from src in hive-2.3.4,but > it falls with the ClassCastException: > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer > Steps to reproduce: > 1）hive.execution.engine is the default value mr > 2）hive.security.authorization.enabled is set to true, and > hive.security.authorization.manager is set to > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider > 3）start hivecli to run command：explain select * from src > I debug the code and find the issue HIVE-18778 causing the above > ClassCastException.If I set hive.in.test to true，the explain command can be > successfully executed。 > Now,I have one question,due to hive.in.test cann't be modified at runtime.how > to run explain command with using default authorization in hive-2.3.4, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21489) explain command throw ClassCastException in 2.3.4
[
https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247289#comment-17247289
]
Oleksiy Sayankin edited comment on HIVE-21489 at 12/10/20, 2:53 PM:
[~Ping Lu] [~daijy]
The same error happens when Hive is configured for Storage Base Authorization.
{code}
hive.security.authorization.enabled
true
hive.security.metastore.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.metastore.authenticator.manager
org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.metastore.pre.event.listeners
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
{code}
{code}
FAILED: ClassCastException
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer
{code}
So the patch is valid. Please commit to the master.
was (Author: osayankin):
The same error happens when Hive is configured for Storage Base Authorization.
{code}
hive.security.authorization.enabled
true
hive.security.metastore.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.metastore.authenticator.manager
org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.metastore.pre.event.listeners
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
{code}
{code}
FAILED: ClassCastException
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer
{code}
So the patch is valid. Please commit to the master.
> explain command throw ClassCastException in 2.3.4
> -
>
> Key: HIVE-21489
> URL: https://issues.apache.org/jira/browse/HIVE-21489
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: Ping Lu
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch
>
>
> I'm trying to run commands like explain select * from src in hive-2.3.4,but
> it falls with the ClassCastException:
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer
> Steps to reproduce:
> 1）hive.execution.engine is the default value mr
> 2）hive.security.authorization.enabled is set to true, and
> hive.security.authorization.manager is set to
> org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
> 3）start hivecli to run command：explain select * from src
> I debug the code and find the issue HIVE-18778 causing the above
> ClassCastException.If I set hive.in.test to true，the explain command can be
> successfully executed。
> Now,I have one question,due to hive.in.test cann't be modified at runtime.how
> to run explain command with using default authorization in hive-2.3.4,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21489) explain command throw ClassCastException in 2.3.4
[
https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247289#comment-17247289
]
Oleksiy Sayankin commented on HIVE-21489:
-
The same error happens when Hive is configured for Storage Base Authorization.
{code}
hive.security.authorization.enabled
true
hive.security.metastore.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.metastore.authenticator.manager
org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.metastore.pre.event.listeners
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
{code}
{code}
FAILED: ClassCastException
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer
{code}
So the patch is valid. Please commit to the master.
> explain command throw ClassCastException in 2.3.4
> -
>
> Key: HIVE-21489
> URL: https://issues.apache.org/jira/browse/HIVE-21489
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: Ping Lu
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch
>
>
> I'm trying to run commands like explain select * from src in hive-2.3.4,but
> it falls with the ClassCastException:
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer
> Steps to reproduce:
> 1）hive.execution.engine is the default value mr
> 2）hive.security.authorization.enabled is set to true, and
> hive.security.authorization.manager is set to
> org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
> 3）start hivecli to run command：explain select * from src
> I debug the code and find the issue HIVE-18778 causing the above
> ClassCastException.If I set hive.in.test to true，the explain command can be
> successfully executed。
> Now,I have one question,due to hive.in.test cann't be modified at runtime.how
> to run explain command with using default authorization in hive-2.3.4,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21843) UNION query with regular expressions for column name does not work
[
https://issues.apache.org/jira/browse/HIVE-21843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241432#comment-17241432
]
Oleksiy Sayankin commented on HIVE-21843:
-
Thanks
> UNION query with regular expressions for column name does not work
> --
>
> Key: HIVE-21843
> URL: https://issues.apache.org/jira/browse/HIVE-21843
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-21843.1.patch, HIVE-21843.10.patch,
> HIVE-21843.4.patch, HIVE-21843.6.patch, HIVE-21843.7.patch,
> HIVE-21843.8.patch, HIVE-21843.9.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *STEPS TO REPRODUCE:*
> 1. Create a table:
> {code:java}CREATE TABLE t (a1 INT, a2 INT);
> INSERT INTO TABLE t VALUES (1,1),(1,2),(2,1),(2,2);{code}
> 2. SET hive.support.quoted.identifiers to "none":
> {code:java}SET hive.support.quoted.identifiers=none;{code}
> 3. Run the query:
> {code:java}SELECT `(a1)?+.+` FROM t
> UNION
> SELECT `(a2)?+.+` FROM t;{code}
> *ACTUAL RESULT:*
> The query fails with an exception:
> {code:java}2019-05-23T01:36:53,604 ERROR
> [9aa457a9-1c74-466e-abef-ec2f007117f3 main] ql.Driver: FAILED:
> SemanticException Line 0:-1 Invalid column reference '`(a1)?+.+`'
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid column
> reference '`(a1)?+.+`'
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11734)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11674)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11642)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11620)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5225)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6330)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9659)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10579)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10457)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:11202)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:481)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11215)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:774)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:697)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
> FYI: [~sershe]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237685#comment-17237685
]
Oleksiy Sayankin commented on HIVE-18728:
-
[~hunterl] wrote:
{quote}
Unfortunately there's no way for me to change the PR on github to your
authorship, which is why I suggested you open your own PR and I close mine. If
you do chose to do this just leave a comment here or on the github link I
posted above and I'll be sure to close mine out.
{quote}
You can easily change the ownership if existing commit is on top of the PR.
{code}
git commit --amend --author="Name "
git push origin +master --force
{code}
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237684#comment-17237684
]
Oleksiy Sayankin commented on HIVE-18728:
-
Created https://github.com/apache/hive/pull/1699
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237553#comment-17237553
]
Oleksiy Sayankin commented on HIVE-18728:
-
I have no enough permissions to create a PR in Hive so that's why I added the
files with patches. If you are able to set me as an author in the PR, then the
rest is ok from my side.
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Hunter Logan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237539#comment-17237539
]
Oleksiy Sayankin commented on HIVE-18728:
-
For some reason my solution was ignored by the community. I am still here and I
wish to commit this to Hive
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Hunter Logan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Attachment: HIVE-23075.2.patch > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch, HIVE-23075.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Status: Patch Available (was: In Progress) > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch, HIVE-23075.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Status: In Progress (was: Patch Available) > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23075) Add property for manual configuration of SSL version
[
https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1700#comment-1700
]
Oleksiy Sayankin commented on HIVE-23075:
-
*FIXED*
*SOLUTION*
*1.*
Add new property {{hive.ssl.protocol.version}} with default value {{TLSv1.2}}.
This is SSL protocol versions for all Hive Servers. This property is set in
{{hive-site.xml}} and requires Hive services to be restarted to make the change
take effect.
*2.*
Add logging with SSL version. See example in HiveServer2:
{code}
2020-03-23T14:24:57,907 INFO [main] http.HttpServer: Current SSL protocol
version is TLSv1.2
2020-03-23T14:24:58,008 INFO [Thread-8] auth.HiveAuthUtils: SSL Server Socket
Enabled Protocols: [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]
2020-03-23T14:24:58,008 INFO [Thread-8] thrift.ThriftCLIService: Current SSL
protocol version is TLSv1.2
2020-03-23T14:24:58,119 INFO [main] server.AbstractConnector: Started
ServerConnector@71978f46{SSL,[ssl, http/1.1]}{0.0.0.0:10002}
{code}
In webhcat:
{code}
INFO | 23 Mar 2020 14:25:03,363 | org.apache.hive.hcatalog.templeton.Main |
Using SSL for templeton.
INFO | 23 Mar 2020 14:25:03,641 | org.apache.hive.hcatalog.templeton.Main |
Current SSL protocol version is TLSv1.2
{code}
*EFFECTS*
1. JDBC SSL connection
2. WebHCat SSL conection
> Add property for manual configuration of SSL version
>
>
> Key: HIVE-23075
> URL: https://issues.apache.org/jira/browse/HIVE-23075
> Project: Hive
> Issue Type: Improvement
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-23075.1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Add property for manual configuration of SSL version
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Attachment: HIVE-23075.1.patch > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Status: Patch Available (was: In Progress) > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23075 started by Oleksiy Sayankin. --- > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-23075: --- > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Status: Patch Available (was: In Progress)
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch,
> HIVE-22980.3.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Status: In Progress (was: Patch Available)
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch,
> HIVE-22980.3.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Attachment: HIVE-22980.3.patch
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch,
> HIVE-22980.3.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Attachment: HIVE-22980.2.patch
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Status: In Progress (was: Patch Available)
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Status: Patch Available (was: In Progress)
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052071#comment-17052071
]
Oleksiy Sayankin commented on HIVE-22980:
-
*FIXED*
*SOLUTION*
Add processing of {{mapreduce.input.pathFilter.class}} property in
{{AcidUtils}} for ORC tables. E.g. to enable custom filter:
\\
\\
1. Implement {{CustomPathFilter}}:
{code}
class CustomPathFilter implements PathFilter{
@Override
public boolean accept(Path path) {
String name = path.getName();
return name.startsWith("a");
}
}
{code}
2. Add {{CustomPathFilter}} to configuration.
{code}
PathFilter customPathFilter = new CustomPathFilter();
Configuration conf = new Configuration();
conf.setClass("mapreduce.input.pathFilter.class", customPathFilter.getClass(),
PathFilter.class);
{code}
3. Pass {{Configuration}} to Hive:
{code}
AcidUtils.Directory dir = AcidUtils.getAcidState(fs, new MockPath(fs,
"/tbl/part1"), conf, new ValidReaderWriteIdList("tbl:100:" + Long.MAX_VALUE +
":"), null, false, null, false);
{code}
*EFFECTS*
ORC processing.
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Status: Patch Available (was: In Progress)
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Attachment: HIVE-22980.1.patch
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22980 started by Oleksiy Sayankin.
---
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-22980:
---
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Attachment: HIVE-22919.6.patch
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch, HIVE-22919.6.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: Patch Available (was: In Progress)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch, HIVE-22919.6.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: In Progress (was: Patch Available)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch, HIVE-22919.6.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: In Progress (was: Patch Available)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: Patch Available (was: In Progress)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Attachment: HIVE-22919.5.patch
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Attachment: HIVE-22919.4.patch
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch, HIVE-22919.4.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: Patch Available (was: In Progress)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch, HIVE-22919.4.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: In Progress (was: Patch Available)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041814#comment-17041814
]
Oleksiy Sayankin edited comment on HIVE-22919 at 2/25/20 10:39 AM:
---
*FIXED*
*ROOT-CAUSE*
The root-cause of the issue does not relate to Storage Base Authorization, it
is about correct update of Hive variable {{hive.metastore.warehouse.dir}}. As
one can see from the exception:
{code}
FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
{code}
Hive wants to create new database at {{hdfs:/tmp/m2}} despite the operator
{{SET}} that was executed before
{code}
SET hive.metastore.warehouse.dir=/tmp/m3;
{code}
This happens because {{StorageBasedAuthorizationProvider}} has an instance of
{{Warehouse}} object that has value for {{hive.metastore.warehouse.dir}}. When
a user updates the {{hive.metastore.warehouse.dir}} in {{Configuration}}
instance, this action does not force {{Warehouse}} object to refresh the value
of {{hive.metastore.warehouse.dir}} and hence it has the old one.
*SOLUTION*
Add {{isWarehouseChanged()}} method to check whether
{{hive.metastore.warehouse.dir}} has been changed and recreate {{Warehouse}} in
{{StorageBasedAuthorizationProvider}} if yes.
*EFFECTS*
{{StorageBasedAuthorizationProvider}} initialization.
was (Author: osayankin):
*FIXED*
*ROOT-CAUSE*
The root-cause of the issue does not relate to Storage Base Authorization, it
is about correct update of Hive variable {{hive.metastore.warehouse.dir}}. As
one can see from the exception:
{code}
FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
{code}
Hive wants to create new database at {{hdfs:/tmp/m2}} despite the operator
{{SET}} that was executed before
{code}
SET hive.metastore.warehouse.dir=/tmp/m3;
{code}
This happens because {{StorageBasedAuthorizationProvider}} has an instance of
{{Warehouse}} object that has value for {{hive.metastore.warehouse.dir}}. When
a user updates the {{hive.metastore.warehouse.dir}} in {{HiveConf}} instance,
this action does not force {{Warehouse}} object to refresh the value of
{{hive.metastore.warehouse.dir}} and hence it has the old one.
*SOLUTION*
Add {{isWarehouseChanged()}} method to check whether
{{hive.metastore.warehouse.dir}} has been changed and recreate {{Warehouse}} in
{{StorageBasedAuthorizationProvider}} if yes.
*EFFECTS*
{{StorageBasedAuthorizationProvider}} initialization.
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: Patch Available (was: In Progress)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Attachment: HIVE-22919.3.patch
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch,
> HIVE-22919.3.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: In Progress (was: Patch Available)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043449#comment-17043449
]
Oleksiy Sayankin commented on HIVE-22919:
-
Hi [~kgyrtkirk].
I understand your idea. This line
{code}
hive [...] --hiveconf hive.metastore.warehouse.dir=/tmp/m2
{code}
is only an example. Customer wants to do following in CLI:
{code}
SET hive.metastore.warehouse.dir=/tmp/path1;
SET hive.metastore.warehouse.dir=/tmp/path2;
SET hive.metastore.warehouse.dir=/tmp/path3;
...
-- some other value for hive.metastore.warehouse.dir
{code}
and so on in one CLI session. And this is real use-case from real customer and
they ask me to fix it.
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: In Progress (was: Patch Available)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: Patch Available (was: In Progress)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Attachment: HIVE-22919.2.patch
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041814#comment-17041814
]
Oleksiy Sayankin commented on HIVE-22919:
-
*FIXED*
*ROOT-CAUSE*
The root-cause of the issue does not relate to Storage Base Authorization, it
is about correct update of Hive variable {{hive.metastore.warehouse.dir}}. As
one can see from the exception:
{code}
FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
{code}
Hive wants to create new database at {{hdfs:/tmp/m2}} despite the operator
{{SET}} that was executed before
{code}
SET hive.metastore.warehouse.dir=/tmp/m3;
{code}
This happens because {{StorageBasedAuthorizationProvider}} has an instance of
{{Warehouse}} object that has value for {{hive.metastore.warehouse.dir}}. When
a user updates the {{hive.metastore.warehouse.dir}} in {{HiveConf}} instance,
this action does not force {{Warehouse}} object to refresh the value of
{{hive.metastore.warehouse.dir}} and hence it has the old one.
*SOLUTION*
Add {{isWarehouseChanged()}} method to check whether
{{hive.metastore.warehouse.dir}} has been changed and recreate {{Warehouse}} in
{{StorageBasedAuthorizationProvider}} if yes.
*EFFECTS*
{{StorageBasedAuthorizationProvider}} initialization.
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Status: Patch Available (was: In Progress)
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Attachment: HIVE-22919.1.patch
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22919.1.patch
>
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]$
> {code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}
> sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1
> {code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;
> {code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}
> hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>
> {code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Description:
*ENVIRONMENT:*
Hive-2.3
*STEPS TO REPRODUCE:*
1. Configure Storage Based Authorization:
{code:xml}
hive.security.authorization.enabled
true
hive.security.metastore.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.metastore.authenticator.manager
org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.metastore.pre.event.listeners
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
{code}
2. Create a few directories, change owners and permissions to it:
{code:java}hadoop fs -mkdir /tmp/m1
hadoop fs -mkdir /tmp/m2
hadoop fs -mkdir /tmp/m3
hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
hadoop fs -chmod 700 /tmp/m[1-3]{code}
3. Check permissions:
{code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
[test@node2 ~]$
{code}
4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
{code:java}
sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
hive.metastore.warehouse.dir=/tmp/m1
{code}
5. Perform the next steps:
{code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
SET hive.metastore.warehouse.dir;
-- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
does not have an access:
SET hive.metastore.warehouse.dir=/tmp/m2;
-- 3. Try to create a database:
CREATE DATABASE m2;
-- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
has an access:
SET hive.metastore.warehouse.dir=/tmp/m3;
-- 5. Try to create a database:
CREATE DATABASE m3;
{code}
*ACTUAL RESULT:*
Query 5 fails with an exception below. It does not handle
"hive.metastore.warehouse.dir" proprty:
{code:java}
hive> -- 5. Try to create a database:
hive> CREATE DATABASE m3;
FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
hive>
{code}
*EXPECTED RESULT:*
Query 5 creates a database;
was:
*ENVIRONMENT:*
Hive-2.3
*STEPS TO REPRODUCE:*
1. Configure Storage Based Authorization:
{code:xml}
hive.security.authorization.enabled
true
hive.security.metastore.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.metastore.authenticator.manager
org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.metastore.pre.event.listeners
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
{code}
2. Create a few directories, change owners and permissions to it:
{code:java}hadoop fs -mkdir /tmp/m1
hadoop fs -mkdir /tmp/m2
hadoop fs -mkdir /tmp/m3
hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
hadoop fs -chmod 700 /tmp/m[1-3]{code}
3. Check permissions:
{code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
[test@node2 ~]$
{code}
4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
{code:java}
sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
hive.metastore.warehouse.dir=/tmp/m1
{code}
5. Perform the next steps:
{code:sql
}-- 1. Check "hive.metastore.warehouse.dir" value:
SET hive.metastore.warehouse.dir;
-- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
does not have an access:
SET hive.metastore.warehouse.dir=/tmp/m2;
-- 3. Try to create a database:
CREATE DATABASE m2;
-- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
has an access:
SET hive.metastore.warehouse.dir=/tmp/m3;
-- 5. Try to create a database:
CREATE DATABASE m3;
{code}
*ACTUAL RESULT:*
Query 5 fails with an exception below. It does not handle
"hive.metastore.warehouse.dir" proprty:
{code:java}
hive> -- 5. Try to create a database:
hive> CREATE DATABASE m3;
FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
hive>
{code}
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22919:
Description:
*ENVIRONMENT:*
Hive-2.3
*STEPS TO REPRODUCE:*
1. Configure Storage Based Authorization:
{code:xml}
hive.security.authorization.enabled
true
hive.security.metastore.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.metastore.authenticator.manager
org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.metastore.pre.event.listeners
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
{code}
2. Create a few directories, change owners and permissions to it:
{code:java}hadoop fs -mkdir /tmp/m1
hadoop fs -mkdir /tmp/m2
hadoop fs -mkdir /tmp/m3
hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
hadoop fs -chmod 700 /tmp/m[1-3]{code}
3. Check permissions:
{code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
[test@node2 ~]$
{code}
4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
{code:java}
sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
hive.metastore.warehouse.dir=/tmp/m1
{code}
5. Perform the next steps:
{code:sql
}-- 1. Check "hive.metastore.warehouse.dir" value:
SET hive.metastore.warehouse.dir;
-- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
does not have an access:
SET hive.metastore.warehouse.dir=/tmp/m2;
-- 3. Try to create a database:
CREATE DATABASE m2;
-- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
has an access:
SET hive.metastore.warehouse.dir=/tmp/m3;
-- 5. Try to create a database:
CREATE DATABASE m3;
{code}
*ACTUAL RESULT:*
Query 5 fails with an exception below. It does not handle
"hive.metastore.warehouse.dir" proprty:
{code:java}
hive> -- 5. Try to create a database:
hive> CREATE DATABASE m3;
FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
hive>
{code}
*EXPECTED RESULT:*
Query 5 creates a database;
was:
*ENVIRONMENT:*
Hive-2.3
*STEPS TO REPRODUCE:*
1. Configure Storage Based Authorization:
{code:xml}
hive.security.authorization.enabled
true
hive.security.metastore.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.metastore.authenticator.manager
org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.metastore.pre.event.listeners
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
{code}
2. Create a few directories, change owners and permissions to it:
{code:java}hadoop fs -mkdir /tmp/m1
hadoop fs -mkdir /tmp/m2
hadoop fs -mkdir /tmp/m3
hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
hadoop fs -chmod 700 /tmp/m[1-3]{code}
3. Check permissions:
{code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
[test@node2 ~]${code}
4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
{code:java}sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
hive.metastore.warehouse.dir=/tmp/m1{code}
5. Perform the next steps:
{code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
SET hive.metastore.warehouse.dir;
-- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
does not have an access:
SET hive.metastore.warehouse.dir=/tmp/m2;
-- 3. Try to create a database:
CREATE DATABASE m2;
-- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
has an access:
SET hive.metastore.warehouse.dir=/tmp/m3;
-- 5. Try to create a database:
CREATE DATABASE m3;{code}
*ACTUAL RESULT:*
Query 5 fails with an exception below. It does not handle
"hive.metastore.warehouse.dir" proprty:
{code:java}hive> -- 5. Try to create a database:
hive> CREATE DATABASE m3;
FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
hive>{code}
*EXPECTED
[jira] [Work started] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22919 started by Oleksiy Sayankin.
---
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]${code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1{code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;{code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>{code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[
https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-22919:
---
> StorageBasedAuthorizationProvider does not allow create databases after
> changing hive.metastore.warehouse.dir
> -
>
> Key: HIVE-22919
> URL: https://issues.apache.org/jira/browse/HIVE-22919
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
>
> *ENVIRONMENT:*
> Hive-2.3
> *STEPS TO REPRODUCE:*
> 1. Configure Storage Based Authorization:
> {code:xml}
> hive.security.authorization.enabled
> true
>
>
> hive.security.metastore.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.authorization.manager
>
> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
>
>
> hive.security.metastore.authenticator.manager
>
> org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
>
>
> hive.metastore.pre.event.listeners
>
> org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
> {code}
> 2. Create a few directories, change owners and permissions to it:
> {code:java}hadoop fs -mkdir /tmp/m1
> hadoop fs -mkdir /tmp/m2
> hadoop fs -mkdir /tmp/m3
> hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
> hadoop fs -chmod 700 /tmp/m[1-3]{code}
> 3. Check permissions:
> {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
> drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
> drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
> drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
> [test@node2 ~]${code}
> 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
> with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
> {code:java}sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
> hive.metastore.warehouse.dir=/tmp/m1{code}
> 5. Perform the next steps:
> {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
> SET hive.metastore.warehouse.dir;
> -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user does not have an access:
> SET hive.metastore.warehouse.dir=/tmp/m2;
> -- 3. Try to create a database:
> CREATE DATABASE m2;
> -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1"
> user has an access:
> SET hive.metastore.warehouse.dir=/tmp/m3;
> -- 5. Try to create a database:
> CREATE DATABASE m3;{code}
> *ACTUAL RESULT:*
> Query 5 fails with an exception below. It does not handle
> "hive.metastore.warehouse.dir" proprty:
> {code:java}hive> -- 5. Try to create a database:
> hive> CREATE DATABASE m3;
> FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
> testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
> hive>{code}
> *EXPECTED RESULT:*
> Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
However most of the entries have {{HL_ACQUIRED_AT=NULL}} in {{HIVE_LOCKS}}:
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null;
+--+
| COUNT(*) |
+--+
| 123437 |
+--+
1 row in set (0.04 sec)
{code}
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null;
+--+
| COUNT(*) |
+--+
| 97 |
+--+
1 row in set (0.04 sec)
{code}
Hot-fix is to remove orphan/lost locks from {{HIVE_LOCKS}}, but this does not
solve the problem in the future.
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE |
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
However most of the entries are {{HL_ACQUIRED_AT=NULL}} in {{HIVE_LOCKS}}:
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null;
+--+
| COUNT(*) |
+--+
| 123437 |
+--+
1 row in set (0.04 sec)
{code}
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null;
+--+
| COUNT(*) |
+--+
| 97 |
+--+
1 row in set (0.04 sec)
{code}
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
|
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
The delete sql on {{HIVE_LOCKS}} is based on column {{HL_ACQUIRED_AT}}, however
most of the entries are {{NULL}}:
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null;
+--+
| COUNT(*) |
+--+
| 123437 |
+--+
1 row in set (0.04 sec)
{code}
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null;
+--+
| COUNT(*) |
+--+
| 97 |
+--+
1 row in set (0.04 sec)
{code}
So after deleting,
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
The delete sql on {{HIVE_LOCKS}} is based on column {{HL_ACQUIRED_AT}}, however
most of the entries are {{NULL}}:
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null;
+--+
| COUNT(*) |
+--+
| 123437 |
+--+
1 row in set (0.04 sec)
{code}
{code}
mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null;
+--+
| COUNT(*) |
+--+
| 97 |
+--+
1 row in set (0.04 sec)
{code}
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 |
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Environment: Hive-2.3.
> Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
> -
>
> Key: HIVE-22911
> URL: https://issues.apache.org/jira/browse/HIVE-22911
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
> Environment: Hive-2.3.
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Critical
>
> We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}.
> There are more than 120k locks in HIVE_LOCKS of MySQL database. We also
> checked the top 3 tables which are related to the existing locks:
>
> {code}
> mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by
> 3 desc limit 10;
> +---+--+--+
> | HL_DB | HL_TABLE | count(*) |
> +---+--+--+
> | db1 | table1 | 66984 |
> | db1 | table2 | 33208 |
> | db1 | table3 | 9315 |
> …
> {code}
> For table “db1. table1”, here are 3 Hive sessions related, and each of the
> Hive session is waiting for 22328 read locks. This is because this table
> “db1. table1” is a huge partition table, and it has more than 200k child
> partitions. I am guessing each of Hive session was trying to do a full table
> scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is
> the list:
>
> {code}
> mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
> dt,count(*) as cnt from HIVE_LOCKS
> -> group by 1 order by 1;
> +++
> | dt | cnt|
> +++
> | 1969-12-31 | 2 |
> | 2019-05-20 | 10 |
> | 2019-05-21 | 3 |
> | 2019-05-23 | 5 |
> | 2019-05-24 | 2 |
> | 2019-05-25 | 1 |
> | 2019-05-29 | 7 |
> | 2019-05-30 | 2 |
> | 2019-06-11 | 13 |
> | 2019-06-28 | 3 |
> | 2019-07-02 | 2 |
> | 2019-07-04 | 5 |
> | 2019-07-09 | 1 |
> | 2019-07-15 | 2 |
> | 2019-07-16 | 1 |
> | 2019-07-18 | 2 |
> | 2019-07-20 | 3 |
> | 2019-07-29 | 5 |
> | 2019-07-30 | 9 |
> | 2019-07-31 | 7 |
> | 2019-08-02 | 2 |
> | 2019-08-06 | 5 |
> | 2019-08-07 | 17 |
> | 2019-08-08 | 8 |
> | 2019-08-09 | 5 |
> | 2019-08-21 | 1 |
> | 2019-08-22 | 20 |
> | 2019-08-23 | 1 |
> | 2019-08-26 | 5 |
> | 2019-08-27 | 98 |
> | 2019-08-28 | 3 |
> | 2019-08-29 | 1 |
> | 2019-09-02 | 3 |
> | 2019-09-04 | 3 |
> | 2019-09-05 |105 |
> | 2019-09-06 | 3 |
> | 2019-09-07 | 2 |
> | 2019-09-09 | 6 |
> | 2019-09-12 | 9 |
> | 2019-09-13 | 1 |
> | 2019-09-17 | 1 |
> | 2019-09-24 | 3 |
> | 2019-09-26 | 6 |
> | 2019-09-27 | 4 |
> | 2019-09-30 | 1 |
> | 2019-10-01 | 2 |
> | 2019-10-03 | 9 |
> | 2019-10-04 | 2 |
> | 2019-10-06 | 1 |
> | 2019-10-08 | 1 |
> | 2019-10-09 | 1 |
> | 2019-10-10 | 6 |
> | 2019-10-11 | 1 |
> | 2019-10-16 | 13 |
> | 2019-10-17 | 1 |
> | 2019-10-18 | 2 |
> | 2019-10-19 | 2 |
> | 2019-10-21 | 10 |
> | 2019-10-22 | 6 |
> | 2019-10-28 | 2 |
> | 2019-10-29 | 4 |
> | 2019-10-30 | 2 |
> | 2019-10-31 | 2 |
> | 2019-11-05 | 2 |
> | 2019-11-06 | 2 |
> | 2019-11-11 | 1 |
> | 2019-11-13 | 1 |
> | 2019-11-14 | 1 |
> | 2019-11-21 | 4 |
> | 2019-11-26 | 1 |
> | 2019-11-27 | 1 |
> | 2019-12-05 | 4 |
> | 2019-12-06 | 2 |
> | 2019-12-12 | 1 |
> | 2019-12-14 | 1 |
> | 2019-12-15 | 3 |
> | 2019-12-16 | 1 |
> | 2019-12-17 | 1 |
> | 2019-12-18 | 1 |
> | 2019-12-19 | 2 |
> | 2019-12-20 | 2 |
> | 2019-12-23 | 1 |
> | 2019-12-27 | 1 |
> | 2020-01-07 | 1 |
> | 2020-01-08 | 14 |
> | 2020-01-09 | 2 |
> | 2020-01-12 |372 |
> | 2020-01-14 | 2 |
> | 2020-01-15 | 1 |
> | 2020-01-20 | 11 |
> | 2020-01-21 | 119253 |
> | 2020-01-23 |113 |
> | 2020-01-24 | 4 |
> | 2020-01-25 |536 |
> | 2020-01-26 | 2132 |
> | 2020-01-27 |396 |
> | 2020-01-28 | 1 |
> | 2020-01-29 | 3 |
> | 2020-01-30 | 11 |
> | 2020-01-31 | 11 |
> | 2020-02-03 | 2 |
> | 2020-02-04 | 4 |
> | 2020-02-05 | 5 |
> | 2020-02-06 | 8 |
> | 2020-02-10 | 32 |
> | 2020-02-11 | 15 |
> | 2020-02-12 | 14 |
> | 2020-02-13 | 1 |
> | 2020-02-14 | 92 |
> +++
> 109 rows in set (0.16 sec)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Component/s: Transactions
> Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
> -
>
> Key: HIVE-22911
> URL: https://issues.apache.org/jira/browse/HIVE-22911
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Critical
>
> We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}.
> There are more than 120k locks in HIVE_LOCKS of MySQL database. We also
> checked the top 3 tables which are related to the existing locks:
>
> {code}
> mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by
> 3 desc limit 10;
> +---+--+--+
> | HL_DB | HL_TABLE | count(*) |
> +---+--+--+
> | db1 | table1 | 66984 |
> | db1 | table2 | 33208 |
> | db1 | table3 | 9315 |
> …
> {code}
> For table “db1. table1”, here are 3 Hive sessions related, and each of the
> Hive session is waiting for 22328 read locks. This is because this table
> “db1. table1” is a huge partition table, and it has more than 200k child
> partitions. I am guessing each of Hive session was trying to do a full table
> scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is
> the list:
>
> {code}
> mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
> dt,count(*) as cnt from HIVE_LOCKS
> -> group by 1 order by 1;
> +++
> | dt | cnt|
> +++
> | 1969-12-31 | 2 |
> | 2019-05-20 | 10 |
> | 2019-05-21 | 3 |
> | 2019-05-23 | 5 |
> | 2019-05-24 | 2 |
> | 2019-05-25 | 1 |
> | 2019-05-29 | 7 |
> | 2019-05-30 | 2 |
> | 2019-06-11 | 13 |
> | 2019-06-28 | 3 |
> | 2019-07-02 | 2 |
> | 2019-07-04 | 5 |
> | 2019-07-09 | 1 |
> | 2019-07-15 | 2 |
> | 2019-07-16 | 1 |
> | 2019-07-18 | 2 |
> | 2019-07-20 | 3 |
> | 2019-07-29 | 5 |
> | 2019-07-30 | 9 |
> | 2019-07-31 | 7 |
> | 2019-08-02 | 2 |
> | 2019-08-06 | 5 |
> | 2019-08-07 | 17 |
> | 2019-08-08 | 8 |
> | 2019-08-09 | 5 |
> | 2019-08-21 | 1 |
> | 2019-08-22 | 20 |
> | 2019-08-23 | 1 |
> | 2019-08-26 | 5 |
> | 2019-08-27 | 98 |
> | 2019-08-28 | 3 |
> | 2019-08-29 | 1 |
> | 2019-09-02 | 3 |
> | 2019-09-04 | 3 |
> | 2019-09-05 |105 |
> | 2019-09-06 | 3 |
> | 2019-09-07 | 2 |
> | 2019-09-09 | 6 |
> | 2019-09-12 | 9 |
> | 2019-09-13 | 1 |
> | 2019-09-17 | 1 |
> | 2019-09-24 | 3 |
> | 2019-09-26 | 6 |
> | 2019-09-27 | 4 |
> | 2019-09-30 | 1 |
> | 2019-10-01 | 2 |
> | 2019-10-03 | 9 |
> | 2019-10-04 | 2 |
> | 2019-10-06 | 1 |
> | 2019-10-08 | 1 |
> | 2019-10-09 | 1 |
> | 2019-10-10 | 6 |
> | 2019-10-11 | 1 |
> | 2019-10-16 | 13 |
> | 2019-10-17 | 1 |
> | 2019-10-18 | 2 |
> | 2019-10-19 | 2 |
> | 2019-10-21 | 10 |
> | 2019-10-22 | 6 |
> | 2019-10-28 | 2 |
> | 2019-10-29 | 4 |
> | 2019-10-30 | 2 |
> | 2019-10-31 | 2 |
> | 2019-11-05 | 2 |
> | 2019-11-06 | 2 |
> | 2019-11-11 | 1 |
> | 2019-11-13 | 1 |
> | 2019-11-14 | 1 |
> | 2019-11-21 | 4 |
> | 2019-11-26 | 1 |
> | 2019-11-27 | 1 |
> | 2019-12-05 | 4 |
> | 2019-12-06 | 2 |
> | 2019-12-12 | 1 |
> | 2019-12-14 | 1 |
> | 2019-12-15 | 3 |
> | 2019-12-16 | 1 |
> | 2019-12-17 | 1 |
> | 2019-12-18 | 1 |
> | 2019-12-19 | 2 |
> | 2019-12-20 | 2 |
> | 2019-12-23 | 1 |
> | 2019-12-27 | 1 |
> | 2020-01-07 | 1 |
> | 2020-01-08 | 14 |
> | 2020-01-09 | 2 |
> | 2020-01-12 |372 |
> | 2020-01-14 | 2 |
> | 2020-01-15 | 1 |
> | 2020-01-20 | 11 |
> | 2020-01-21 | 119253 |
> | 2020-01-23 |113 |
> | 2020-01-24 | 4 |
> | 2020-01-25 |536 |
> | 2020-01-26 | 2132 |
> | 2020-01-27 |396 |
> | 2020-01-28 | 1 |
> | 2020-01-29 | 3 |
> | 2020-01-30 | 11 |
> | 2020-01-31 | 11 |
> | 2020-02-03 | 2 |
> | 2020-02-04 | 4 |
> | 2020-02-05 | 5 |
> | 2020-02-06 | 8 |
> | 2020-02-10 | 32 |
> | 2020-02-11 | 15 |
> | 2020-02-12 | 14 |
> | 2020-02-13 | 1 |
> | 2020-02-14 | 92 |
> +++
> 109 rows in set (0.16 sec)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Component/s: Locking
> Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
> -
>
> Key: HIVE-22911
> URL: https://issues.apache.org/jira/browse/HIVE-22911
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Critical
>
> We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}.
> There are more than 120k locks in HIVE_LOCKS of MySQL database. We also
> checked the top 3 tables which are related to the existing locks:
>
> {code}
> mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by
> 3 desc limit 10;
> +---+--+--+
> | HL_DB | HL_TABLE | count(*) |
> +---+--+--+
> | db1 | table1 | 66984 |
> | db1 | table2 | 33208 |
> | db1 | table3 | 9315 |
> …
> {code}
> For table “db1. table1”, here are 3 Hive sessions related, and each of the
> Hive session is waiting for 22328 read locks. This is because this table
> “db1. table1” is a huge partition table, and it has more than 200k child
> partitions. I am guessing each of Hive session was trying to do a full table
> scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is
> the list:
>
> {code}
> mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
> dt,count(*) as cnt from HIVE_LOCKS
> -> group by 1 order by 1;
> +++
> | dt | cnt|
> +++
> | 1969-12-31 | 2 |
> | 2019-05-20 | 10 |
> | 2019-05-21 | 3 |
> | 2019-05-23 | 5 |
> | 2019-05-24 | 2 |
> | 2019-05-25 | 1 |
> | 2019-05-29 | 7 |
> | 2019-05-30 | 2 |
> | 2019-06-11 | 13 |
> | 2019-06-28 | 3 |
> | 2019-07-02 | 2 |
> | 2019-07-04 | 5 |
> | 2019-07-09 | 1 |
> | 2019-07-15 | 2 |
> | 2019-07-16 | 1 |
> | 2019-07-18 | 2 |
> | 2019-07-20 | 3 |
> | 2019-07-29 | 5 |
> | 2019-07-30 | 9 |
> | 2019-07-31 | 7 |
> | 2019-08-02 | 2 |
> | 2019-08-06 | 5 |
> | 2019-08-07 | 17 |
> | 2019-08-08 | 8 |
> | 2019-08-09 | 5 |
> | 2019-08-21 | 1 |
> | 2019-08-22 | 20 |
> | 2019-08-23 | 1 |
> | 2019-08-26 | 5 |
> | 2019-08-27 | 98 |
> | 2019-08-28 | 3 |
> | 2019-08-29 | 1 |
> | 2019-09-02 | 3 |
> | 2019-09-04 | 3 |
> | 2019-09-05 |105 |
> | 2019-09-06 | 3 |
> | 2019-09-07 | 2 |
> | 2019-09-09 | 6 |
> | 2019-09-12 | 9 |
> | 2019-09-13 | 1 |
> | 2019-09-17 | 1 |
> | 2019-09-24 | 3 |
> | 2019-09-26 | 6 |
> | 2019-09-27 | 4 |
> | 2019-09-30 | 1 |
> | 2019-10-01 | 2 |
> | 2019-10-03 | 9 |
> | 2019-10-04 | 2 |
> | 2019-10-06 | 1 |
> | 2019-10-08 | 1 |
> | 2019-10-09 | 1 |
> | 2019-10-10 | 6 |
> | 2019-10-11 | 1 |
> | 2019-10-16 | 13 |
> | 2019-10-17 | 1 |
> | 2019-10-18 | 2 |
> | 2019-10-19 | 2 |
> | 2019-10-21 | 10 |
> | 2019-10-22 | 6 |
> | 2019-10-28 | 2 |
> | 2019-10-29 | 4 |
> | 2019-10-30 | 2 |
> | 2019-10-31 | 2 |
> | 2019-11-05 | 2 |
> | 2019-11-06 | 2 |
> | 2019-11-11 | 1 |
> | 2019-11-13 | 1 |
> | 2019-11-14 | 1 |
> | 2019-11-21 | 4 |
> | 2019-11-26 | 1 |
> | 2019-11-27 | 1 |
> | 2019-12-05 | 4 |
> | 2019-12-06 | 2 |
> | 2019-12-12 | 1 |
> | 2019-12-14 | 1 |
> | 2019-12-15 | 3 |
> | 2019-12-16 | 1 |
> | 2019-12-17 | 1 |
> | 2019-12-18 | 1 |
> | 2019-12-19 | 2 |
> | 2019-12-20 | 2 |
> | 2019-12-23 | 1 |
> | 2019-12-27 | 1 |
> | 2020-01-07 | 1 |
> | 2020-01-08 | 14 |
> | 2020-01-09 | 2 |
> | 2020-01-12 |372 |
> | 2020-01-14 | 2 |
> | 2020-01-15 | 1 |
> | 2020-01-20 | 11 |
> | 2020-01-21 | 119253 |
> | 2020-01-23 |113 |
> | 2020-01-24 | 4 |
> | 2020-01-25 |536 |
> | 2020-01-26 | 2132 |
> | 2020-01-27 |396 |
> | 2020-01-28 | 1 |
> | 2020-01-29 | 3 |
> | 2020-01-30 | 11 |
> | 2020-01-31 | 11 |
> | 2020-02-03 | 2 |
> | 2020-02-04 | 4 |
> | 2020-02-05 | 5 |
> | 2020-02-06 | 8 |
> | 2020-02-10 | 32 |
> | 2020-02-11 | 15 |
> | 2020-02-12 | 14 |
> | 2020-02-13 | 1 |
> | 2020-02-14 | 92 |
> +++
> 109 rows in set (0.16 sec)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22911 started by Oleksiy Sayankin.
---
> Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
> -
>
> Key: HIVE-22911
> URL: https://issues.apache.org/jira/browse/HIVE-22911
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Critical
>
> We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}.
> There are more than 120k locks in HIVE_LOCKS of MySQL database. We also
> checked the top 3 tables which are related to the existing locks:
>
> {code}
> mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by
> 3 desc limit 10;
> +---+--+--+
> | HL_DB | HL_TABLE | count(*) |
> +---+--+--+
> | db1 | table1 | 66984 |
> | db1 | table2 | 33208 |
> | db1 | table3 | 9315 |
> …
> {code}
> For table “db1. table1”, here are 3 Hive sessions related, and each of the
> Hive session is waiting for 22328 read locks. This is because this table
> “db1. table1” is a huge partition table, and it has more than 200k child
> partitions. I am guessing each of Hive session was trying to do a full table
> scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is
> the list:
>
> {code}
> MariaDB [customer]> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as
> date) as dt,count(*) as cnt from HIVE_LOCKS
> -> group by 1 order by 1;
> +++
> | dt | cnt|
> +++
> | 1969-12-31 | 2 |
> | 2019-05-20 | 10 |
> | 2019-05-21 | 3 |
> | 2019-05-23 | 5 |
> | 2019-05-24 | 2 |
> | 2019-05-25 | 1 |
> | 2019-05-29 | 7 |
> | 2019-05-30 | 2 |
> | 2019-06-11 | 13 |
> | 2019-06-28 | 3 |
> | 2019-07-02 | 2 |
> | 2019-07-04 | 5 |
> | 2019-07-09 | 1 |
> | 2019-07-15 | 2 |
> | 2019-07-16 | 1 |
> | 2019-07-18 | 2 |
> | 2019-07-20 | 3 |
> | 2019-07-29 | 5 |
> | 2019-07-30 | 9 |
> | 2019-07-31 | 7 |
> | 2019-08-02 | 2 |
> | 2019-08-06 | 5 |
> | 2019-08-07 | 17 |
> | 2019-08-08 | 8 |
> | 2019-08-09 | 5 |
> | 2019-08-21 | 1 |
> | 2019-08-22 | 20 |
> | 2019-08-23 | 1 |
> | 2019-08-26 | 5 |
> | 2019-08-27 | 98 |
> | 2019-08-28 | 3 |
> | 2019-08-29 | 1 |
> | 2019-09-02 | 3 |
> | 2019-09-04 | 3 |
> | 2019-09-05 |105 |
> | 2019-09-06 | 3 |
> | 2019-09-07 | 2 |
> | 2019-09-09 | 6 |
> | 2019-09-12 | 9 |
> | 2019-09-13 | 1 |
> | 2019-09-17 | 1 |
> | 2019-09-24 | 3 |
> | 2019-09-26 | 6 |
> | 2019-09-27 | 4 |
> | 2019-09-30 | 1 |
> | 2019-10-01 | 2 |
> | 2019-10-03 | 9 |
> | 2019-10-04 | 2 |
> | 2019-10-06 | 1 |
> | 2019-10-08 | 1 |
> | 2019-10-09 | 1 |
> | 2019-10-10 | 6 |
> | 2019-10-11 | 1 |
> | 2019-10-16 | 13 |
> | 2019-10-17 | 1 |
> | 2019-10-18 | 2 |
> | 2019-10-19 | 2 |
> | 2019-10-21 | 10 |
> | 2019-10-22 | 6 |
> | 2019-10-28 | 2 |
> | 2019-10-29 | 4 |
> | 2019-10-30 | 2 |
> | 2019-10-31 | 2 |
> | 2019-11-05 | 2 |
> | 2019-11-06 | 2 |
> | 2019-11-11 | 1 |
> | 2019-11-13 | 1 |
> | 2019-11-14 | 1 |
> | 2019-11-21 | 4 |
> | 2019-11-26 | 1 |
> | 2019-11-27 | 1 |
> | 2019-12-05 | 4 |
> | 2019-12-06 | 2 |
> | 2019-12-12 | 1 |
> | 2019-12-14 | 1 |
> | 2019-12-15 | 3 |
> | 2019-12-16 | 1 |
> | 2019-12-17 | 1 |
> | 2019-12-18 | 1 |
> | 2019-12-19 | 2 |
> | 2019-12-20 | 2 |
> | 2019-12-23 | 1 |
> | 2019-12-27 | 1 |
> | 2020-01-07 | 1 |
> | 2020-01-08 | 14 |
> | 2020-01-09 | 2 |
> | 2020-01-12 |372 |
> | 2020-01-14 | 2 |
> | 2020-01-15 | 1 |
> | 2020-01-20 | 11 |
> | 2020-01-21 | 119253 |
> | 2020-01-23 |113 |
> | 2020-01-24 | 4 |
> | 2020-01-25 |536 |
> | 2020-01-26 | 2132 |
> | 2020-01-27 |396 |
> | 2020-01-28 | 1 |
> | 2020-01-29 | 3 |
> | 2020-01-30 | 11 |
> | 2020-01-31 | 11 |
> | 2020-02-03 | 2 |
> | 2020-02-04 | 4 |
> | 2020-02-05 | 5 |
> | 2020-02-06 | 8 |
> | 2020-02-10 | 32 |
> | 2020-02-11 | 15 |
> | 2020-02-12 | 14 |
> | 2020-02-13 | 1 |
> | 2020-02-14 | 92 |
> +++
> 109 rows in set (0.16 sec)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-22911:
---
> Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
> -
>
> Key: HIVE-22911
> URL: https://issues.apache.org/jira/browse/HIVE-22911
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Critical
>
> We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}.
> There are more than 120k locks in HIVE_LOCKS of MySQL database. We also
> checked the top 3 tables which are related to the existing locks:
>
> {code}
> mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by
> 3 desc limit 10;
> +---+--+--+
> | HL_DB | HL_TABLE | count(*) |
> +---+--+--+
> | db1 | table1 | 66984 |
> | db1 | table2 | 33208 |
> | db1 | table3 | 9315 |
> …
> {code}
> For table “db1. table1”, here are 3 Hive sessions related, and each of the
> Hive session is waiting for 22328 read locks. This is because this table
> “db1. table1” is a huge partition table, and it has more than 200k child
> partitions. I am guessing each of Hive session was trying to do a full table
> scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is
> the list:
>
> {code}
> MariaDB [customer]> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as
> date) as dt,count(*) as cnt from HIVE_LOCKS
> -> group by 1 order by 1;
> +++
> | dt | cnt|
> +++
> | 1969-12-31 | 2 |
> | 2019-05-20 | 10 |
> | 2019-05-21 | 3 |
> | 2019-05-23 | 5 |
> | 2019-05-24 | 2 |
> | 2019-05-25 | 1 |
> | 2019-05-29 | 7 |
> | 2019-05-30 | 2 |
> | 2019-06-11 | 13 |
> | 2019-06-28 | 3 |
> | 2019-07-02 | 2 |
> | 2019-07-04 | 5 |
> | 2019-07-09 | 1 |
> | 2019-07-15 | 2 |
> | 2019-07-16 | 1 |
> | 2019-07-18 | 2 |
> | 2019-07-20 | 3 |
> | 2019-07-29 | 5 |
> | 2019-07-30 | 9 |
> | 2019-07-31 | 7 |
> | 2019-08-02 | 2 |
> | 2019-08-06 | 5 |
> | 2019-08-07 | 17 |
> | 2019-08-08 | 8 |
> | 2019-08-09 | 5 |
> | 2019-08-21 | 1 |
> | 2019-08-22 | 20 |
> | 2019-08-23 | 1 |
> | 2019-08-26 | 5 |
> | 2019-08-27 | 98 |
> | 2019-08-28 | 3 |
> | 2019-08-29 | 1 |
> | 2019-09-02 | 3 |
> | 2019-09-04 | 3 |
> | 2019-09-05 |105 |
> | 2019-09-06 | 3 |
> | 2019-09-07 | 2 |
> | 2019-09-09 | 6 |
> | 2019-09-12 | 9 |
> | 2019-09-13 | 1 |
> | 2019-09-17 | 1 |
> | 2019-09-24 | 3 |
> | 2019-09-26 | 6 |
> | 2019-09-27 | 4 |
> | 2019-09-30 | 1 |
> | 2019-10-01 | 2 |
> | 2019-10-03 | 9 |
> | 2019-10-04 | 2 |
> | 2019-10-06 | 1 |
> | 2019-10-08 | 1 |
> | 2019-10-09 | 1 |
> | 2019-10-10 | 6 |
> | 2019-10-11 | 1 |
> | 2019-10-16 | 13 |
> | 2019-10-17 | 1 |
> | 2019-10-18 | 2 |
> | 2019-10-19 | 2 |
> | 2019-10-21 | 10 |
> | 2019-10-22 | 6 |
> | 2019-10-28 | 2 |
> | 2019-10-29 | 4 |
> | 2019-10-30 | 2 |
> | 2019-10-31 | 2 |
> | 2019-11-05 | 2 |
> | 2019-11-06 | 2 |
> | 2019-11-11 | 1 |
> | 2019-11-13 | 1 |
> | 2019-11-14 | 1 |
> | 2019-11-21 | 4 |
> | 2019-11-26 | 1 |
> | 2019-11-27 | 1 |
> | 2019-12-05 | 4 |
> | 2019-12-06 | 2 |
> | 2019-12-12 | 1 |
> | 2019-12-14 | 1 |
> | 2019-12-15 | 3 |
> | 2019-12-16 | 1 |
> | 2019-12-17 | 1 |
> | 2019-12-18 | 1 |
> | 2019-12-19 | 2 |
> | 2019-12-20 | 2 |
> | 2019-12-23 | 1 |
> | 2019-12-27 | 1 |
> | 2020-01-07 | 1 |
> | 2020-01-08 | 14 |
> | 2020-01-09 | 2 |
> | 2020-01-12 |372 |
> | 2020-01-14 | 2 |
> | 2020-01-15 | 1 |
> | 2020-01-20 | 11 |
> | 2020-01-21 | 119253 |
> | 2020-01-23 |113 |
> | 2020-01-24 | 4 |
> | 2020-01-25 |536 |
> | 2020-01-26 | 2132 |
> | 2020-01-27 |396 |
> | 2020-01-28 | 1 |
> | 2020-01-29 | 3 |
> | 2020-01-30 | 11 |
> | 2020-01-31 | 11 |
> | 2020-02-03 | 2 |
> | 2020-02-04 | 4 |
> | 2020-02-05 | 5 |
> | 2020-02-06 | 8 |
> | 2020-02-10 | 32 |
> | 2020-02-11 | 15 |
> | 2020-02-12 | 14 |
> | 2020-02-13 | 1 |
> | 2020-02-14 | 92 |
> +++
> 109 rows in set (0.16 sec)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[
https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22911:
Description:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
{code}
mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as
dt,count(*) as cnt from HIVE_LOCKS
-> group by 1 order by 1;
+++
| dt | cnt|
+++
| 1969-12-31 | 2 |
| 2019-05-20 | 10 |
| 2019-05-21 | 3 |
| 2019-05-23 | 5 |
| 2019-05-24 | 2 |
| 2019-05-25 | 1 |
| 2019-05-29 | 7 |
| 2019-05-30 | 2 |
| 2019-06-11 | 13 |
| 2019-06-28 | 3 |
| 2019-07-02 | 2 |
| 2019-07-04 | 5 |
| 2019-07-09 | 1 |
| 2019-07-15 | 2 |
| 2019-07-16 | 1 |
| 2019-07-18 | 2 |
| 2019-07-20 | 3 |
| 2019-07-29 | 5 |
| 2019-07-30 | 9 |
| 2019-07-31 | 7 |
| 2019-08-02 | 2 |
| 2019-08-06 | 5 |
| 2019-08-07 | 17 |
| 2019-08-08 | 8 |
| 2019-08-09 | 5 |
| 2019-08-21 | 1 |
| 2019-08-22 | 20 |
| 2019-08-23 | 1 |
| 2019-08-26 | 5 |
| 2019-08-27 | 98 |
| 2019-08-28 | 3 |
| 2019-08-29 | 1 |
| 2019-09-02 | 3 |
| 2019-09-04 | 3 |
| 2019-09-05 |105 |
| 2019-09-06 | 3 |
| 2019-09-07 | 2 |
| 2019-09-09 | 6 |
| 2019-09-12 | 9 |
| 2019-09-13 | 1 |
| 2019-09-17 | 1 |
| 2019-09-24 | 3 |
| 2019-09-26 | 6 |
| 2019-09-27 | 4 |
| 2019-09-30 | 1 |
| 2019-10-01 | 2 |
| 2019-10-03 | 9 |
| 2019-10-04 | 2 |
| 2019-10-06 | 1 |
| 2019-10-08 | 1 |
| 2019-10-09 | 1 |
| 2019-10-10 | 6 |
| 2019-10-11 | 1 |
| 2019-10-16 | 13 |
| 2019-10-17 | 1 |
| 2019-10-18 | 2 |
| 2019-10-19 | 2 |
| 2019-10-21 | 10 |
| 2019-10-22 | 6 |
| 2019-10-28 | 2 |
| 2019-10-29 | 4 |
| 2019-10-30 | 2 |
| 2019-10-31 | 2 |
| 2019-11-05 | 2 |
| 2019-11-06 | 2 |
| 2019-11-11 | 1 |
| 2019-11-13 | 1 |
| 2019-11-14 | 1 |
| 2019-11-21 | 4 |
| 2019-11-26 | 1 |
| 2019-11-27 | 1 |
| 2019-12-05 | 4 |
| 2019-12-06 | 2 |
| 2019-12-12 | 1 |
| 2019-12-14 | 1 |
| 2019-12-15 | 3 |
| 2019-12-16 | 1 |
| 2019-12-17 | 1 |
| 2019-12-18 | 1 |
| 2019-12-19 | 2 |
| 2019-12-20 | 2 |
| 2019-12-23 | 1 |
| 2019-12-27 | 1 |
| 2020-01-07 | 1 |
| 2020-01-08 | 14 |
| 2020-01-09 | 2 |
| 2020-01-12 |372 |
| 2020-01-14 | 2 |
| 2020-01-15 | 1 |
| 2020-01-20 | 11 |
| 2020-01-21 | 119253 |
| 2020-01-23 |113 |
| 2020-01-24 | 4 |
| 2020-01-25 |536 |
| 2020-01-26 | 2132 |
| 2020-01-27 |396 |
| 2020-01-28 | 1 |
| 2020-01-29 | 3 |
| 2020-01-30 | 11 |
| 2020-01-31 | 11 |
| 2020-02-03 | 2 |
| 2020-02-04 | 4 |
| 2020-02-05 | 5 |
| 2020-02-06 | 8 |
| 2020-02-10 | 32 |
| 2020-02-11 | 15 |
| 2020-02-12 | 14 |
| 2020-02-13 | 1 |
| 2020-02-14 | 92 |
+++
109 rows in set (0.16 sec)
{code}
was:
We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There
are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the
top 3 tables which are related to the existing locks:
{code}
mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3
desc limit 10;
+---+--+--+
| HL_DB | HL_TABLE | count(*) |
+---+--+--+
| db1 | table1 | 66984 |
| db1 | table2 | 33208 |
| db1 | table3 | 9315 |
…
{code}
For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive
session is waiting for 22328 read locks. This is because this table “db1.
table1” is a huge partition table, and it has more than 200k child partitions.
I am guessing each of Hive session was trying to do a full table scan on it. I
group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
[jira] [Assigned] (HIVE-7089) StorageBasedAuthorizationProvider fails to allow non-admin users to create databases in writable directories
[
https://issues.apache.org/jira/browse/HIVE-7089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-7089:
--
Assignee: Oleksiy Sayankin
> StorageBasedAuthorizationProvider fails to allow non-admin users to create
> databases in writable directories
>
>
> Key: HIVE-7089
> URL: https://issues.apache.org/jira/browse/HIVE-7089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Craig Condit
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-7089.patch
>
>
> When attempting to create a database with a custom location and using
> hive.security.authorizationmanager=org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider,
> an AccessControlException is generated for the default warehouse location,
> not the location which was given in the create database command.
> {noformat}
> hive> create database test LOCATION '/user/ccondit/test';
>
> Authorization failed:java.security.AccessControlException: action WRITE not
> permitted on path hdfs://example.com:8020/apps/hive/warehouse for user
> ccondit. Use SHOW GRANT to get more details.
> 14/05/19 09:50:59 ERROR ql.Driver: Authorization
> failed:java.security.AccessControlException: action WRITE not permitted on
> path hdfs://example.com:8020/apps/hive/warehouse for user ccondit. Use SHOW
> GRANT to get more details.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-7089) StorageBasedAuthorizationProvider fails to allow non-admin users to create databases in writable directories
[
https://issues.apache.org/jira/browse/HIVE-7089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-7089:
---
Status: In Progress (was: Patch Available)
> StorageBasedAuthorizationProvider fails to allow non-admin users to create
> databases in writable directories
>
>
> Key: HIVE-7089
> URL: https://issues.apache.org/jira/browse/HIVE-7089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Craig Condit
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-7089.patch
>
>
> When attempting to create a database with a custom location and using
> hive.security.authorizationmanager=org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider,
> an AccessControlException is generated for the default warehouse location,
> not the location which was given in the create database command.
> {noformat}
> hive> create database test LOCATION '/user/ccondit/test';
>
> Authorization failed:java.security.AccessControlException: action WRITE not
> permitted on path hdfs://example.com:8020/apps/hive/warehouse for user
> ccondit. Use SHOW GRANT to get more details.
> 14/05/19 09:50:59 ERROR ql.Driver: Authorization
> failed:java.security.AccessControlException: action WRITE not permitted on
> path hdfs://example.com:8020/apps/hive/warehouse for user ccondit. Use SHOW
> GRANT to get more details.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900162#comment-16900162
]
Oleksiy Sayankin commented on HIVE-4605:
Added not null verification for {{finalPaths[idx]}}.
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-4605:
---
Status: Patch Available (was: In Progress)
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.0, 0.13.1, 0.13.0, 0.12.0, 0.11.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-4605:
---
Attachment: HIVE-4605.3.patch
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-4605:
---
Status: In Progress (was: Patch Available)
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.3.0, 0.13.1, 0.13.0, 0.12.0, 0.11.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897390#comment-16897390
]
Oleksiy Sayankin commented on HIVE-4605:
I've re-testes failed tests:
{code}
mvn test -Dtest=TestCliDriver
-Dqfile=orc_llap_counters1.q,orc_llap_counters.q,orc_ppd_basic.q,orc_ppd_schema_evol_3a.q,materialized_view_rewrite_4.q
{code}
The fix does not change their behaviour: they fail even without any changes in
master branch.
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
