[jira] [Created] (HIVE-9828) Semantic analyzer does not capture view parent entity for tables referred in view with union all
Prasad Mujumdar created HIVE-9828:
-
Summary: Semantic analyzer does not capture view parent entity for
tables referred in view with union all
Key: HIVE-9828
URL: https://issues.apache.org/jira/browse/HIVE-9828
Project: Hive
Issue Type: Bug
Components: Parser
Affects Versions: 1.1.0
Reporter: Prasad Mujumdar
Hive compiler adds tables used in a view definition in the input entity list,
with the view as parent entity for the table.
In case of a view with union all query, this is not being done property. For
example,
{noformat}
create view view1 as select t.id from (select tab1.id from db.tab1 union all
select tab2.id from db.tab2 ) t;
{noformat}
This query will capture tab1 and tab2 as read entity without view1 as parent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9625) Delegation tokens for HMS are not renewed
[ https://issues.apache.org/jira/browse/HIVE-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315237#comment-14315237 ] Prasad Mujumdar commented on HIVE-9625: --- [~brocknoland] Are you able to verify the patch on with secure HS2, HMS ? I suspect that just calling getDelegationToken() on error won't help. AFAIR the expired tokens are not auto renewed. There's a GC thread that runs every hour to remove the expired token. You might have to renew or drop the token before re acquiring it. Delegation tokens for HMS are not renewed - Key: HIVE-9625 URL: https://issues.apache.org/jira/browse/HIVE-9625 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-9625.1.patch AFAICT the delegation tokens stored in [HiveSessionImplwithUGI |https://github.com/apache/hive/blob/trunk/service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java#L45] for HMS + Impersonation are never renewed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9541) Update people page with new PMC members
[ https://issues.apache.org/jira/browse/HIVE-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304719#comment-14304719 ] Prasad Mujumdar commented on HIVE-9541: --- Thanks [~thejas] and [~prasanth_j]! Update people page with new PMC members --- Key: HIVE-9541 URL: https://issues.apache.org/jira/browse/HIVE-9541 Project: Hive Issue Type: Improvement Components: Website Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Priority: Trivial Attachments: HIVE-9541.1.patch, HIVE-9541.2.patch Move [~jdere], [~owen.omalley], [~prasanth_j], [~vikram.dixit] and [~szehon] from committer list to PMC list. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9383) Improve schema verification error message
[ https://issues.apache.org/jira/browse/HIVE-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14278112#comment-14278112 ] Prasad Mujumdar commented on HIVE-9383: --- +1 Looks fine to me. Improve schema verification error message - Key: HIVE-9383 URL: https://issues.apache.org/jira/browse/HIVE-9383 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-9383.patch Currently the error message just says the schema found. It should say the schema expected as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8938) Compiler should save the transform URI as input entity
[ https://issues.apache.org/jira/browse/HIVE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8938: -- Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks [~brocknoland] for the review. Compiler should save the transform URI as input entity -- Key: HIVE-8938 URL: https://issues.apache.org/jira/browse/HIVE-8938 Project: Hive Issue Type: Bug Components: Parser, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8938.1.patch, HIVE-8938.2.patch, HIVE-8938.3.patch, HIVE-8938.4.patch Compiler should capture the transform URI as input entity. This would enable better auditing for using transforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8938) Compiler should save the transform URI as input entity
[ https://issues.apache.org/jira/browse/HIVE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8938: -- Labels: (was: TODOC15) Updated the [docs|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution] to list the new config property. Compiler should save the transform URI as input entity -- Key: HIVE-8938 URL: https://issues.apache.org/jira/browse/HIVE-8938 Project: Hive Issue Type: Bug Components: Parser, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8938.1.patch, HIVE-8938.2.patch, HIVE-8938.3.patch, HIVE-8938.4.patch Compiler should capture the transform URI as input entity. This would enable better auditing for using transforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8938) Compiler should save the transform URI as input entity
[ https://issues.apache.org/jira/browse/HIVE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8938: -- Attachment: HIVE-8938.4.patch Updated test out file. The tests nonmr_fetch.q and virtual_column.q fail on trunk without the patch as well. Compiler should save the transform URI as input entity -- Key: HIVE-8938 URL: https://issues.apache.org/jira/browse/HIVE-8938 Project: Hive Issue Type: Bug Components: Parser, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8938.1.patch, HIVE-8938.2.patch, HIVE-8938.3.patch, HIVE-8938.4.patch Compiler should capture the transform URI as input entity. This would enable better auditing for using transforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9176) Delegation token interval should be configurable in HadoopThriftAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-9176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256265#comment-14256265 ] Prasad Mujumdar commented on HIVE-9176: --- +1 The nonmr_fetch and virtual_column seems to be unrelated failures. Delegation token interval should be configurable in HadoopThriftAuthBridge -- Key: HIVE-9176 URL: https://issues.apache.org/jira/browse/HIVE-9176 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-9176.1.patch, HIVE-9176.2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8938) Compiler should save the transform URI as input entity
[ https://issues.apache.org/jira/browse/HIVE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8938: -- Attachment: HIVE-8938.3.patch Compiler should save the transform URI as input entity -- Key: HIVE-8938 URL: https://issues.apache.org/jira/browse/HIVE-8938 Project: Hive Issue Type: Bug Components: Parser, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8938.1.patch, HIVE-8938.2.patch, HIVE-8938.3.patch Compiler should capture the transform URI as input entity. This would enable better auditing for using transforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9176) Delegation token interval should be configurable in HadoopThriftAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-9176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14254153#comment-14254153 ] Prasad Mujumdar commented on HIVE-9176: --- +1 Looks fine to me. This hive.cluster.delegation.token.gc-interval is an advance config option and I guess it's fine to define it locally. Though might still be useful to add it in the documenting in [metastore configuration|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] Delegation token interval should be configurable in HadoopThriftAuthBridge -- Key: HIVE-9176 URL: https://issues.apache.org/jira/browse/HIVE-9176 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-9176.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9158) Multiple LDAP server URLs in hive.server2.authentication.ldap.url
[ https://issues.apache.org/jira/browse/HIVE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252263#comment-14252263 ] Prasad Mujumdar commented on HIVE-9158: --- +1 Looks fine to me. Thanks for figuring this out! I believe this will work with Active Directory as well. [~ngangam] If you found any caveats there, please add a note in the docs. Multiple LDAP server URLs in hive.server2.authentication.ldap.url - Key: HIVE-9158 URL: https://issues.apache.org/jira/browse/HIVE-9158 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.14.0 Reporter: Naveen Gangam Assignee: Naveen Gangam Priority: Minor Labels: TODOC15 Attachments: HIVE-9158.1.patch, LDAPClient.java Support for multiple LDAP servers for failover in the event that one stops responding or is down for maintenance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout for all transports types
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247312#comment-14247312 ] Prasad Mujumdar commented on HIVE-6679: --- [~vgumashta] Apache Thrift has added support for keep alive (THRIFT-2788). The trunk is already upgraded to use Thrift 0.9.2 which has that patch. HiveServer2 should support configurable the server side socket timeout for all transports types --- Key: HIVE-6679 URL: https://issues.apache.org/jira/browse/HIVE-6679 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0, 0.14.0 Reporter: Prasad Mujumdar Assignee: Navis Fix For: 0.14.1 Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, HIVE-6679.3.patch, HIVE-6679.4.patch HiveServer2 should support configurable the server side socket read timeout and TCP keep-alive option. Metastore server already support this (and the so is the old hive server). We now have multiple client connectivity options like Kerberos, Delegation Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The configuration should be applicable to all types (if possible). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8935) Add debug logging around token stores
[ https://issues.apache.org/jira/browse/HIVE-8935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222782#comment-14222782 ] Prasad Mujumdar commented on HIVE-8935: --- Looks fine to me. +1 Add debug logging around token stores - Key: HIVE-8935 URL: https://issues.apache.org/jira/browse/HIVE-8935 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-8935.patch, HIVE-8935.patch It's hard to debug issues related to delegation tokens due to a lack of debug logging. This jira is to add debug logging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8938) Compiler should save the transform URI as input entity
Prasad Mujumdar created HIVE-8938: - Summary: Compiler should save the transform URI as input entity Key: HIVE-8938 URL: https://issues.apache.org/jira/browse/HIVE-8938 Project: Hive Issue Type: Bug Components: Parser, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Compiler should capture the transform URI as input entity. This would enable better auditing for using transforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8938) Compiler should save the transform URI as input entity
[ https://issues.apache.org/jira/browse/HIVE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8938: -- Status: Patch Available (was: Open) Compiler should save the transform URI as input entity -- Key: HIVE-8938 URL: https://issues.apache.org/jira/browse/HIVE-8938 Project: Hive Issue Type: Bug Components: Parser, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8938.1.patch Compiler should capture the transform URI as input entity. This would enable better auditing for using transforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8938) Compiler should save the transform URI as input entity
[ https://issues.apache.org/jira/browse/HIVE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8938: -- Attachment: HIVE-8938.1.patch Compiler should save the transform URI as input entity -- Key: HIVE-8938 URL: https://issues.apache.org/jira/browse/HIVE-8938 Project: Hive Issue Type: Bug Components: Parser, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8938.1.patch Compiler should capture the transform URI as input entity. This would enable better auditing for using transforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8938) Compiler should save the transform URI as input entity
[ https://issues.apache.org/jira/browse/HIVE-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8938: -- Attachment: HIVE-8938.2.patch Compiler should save the transform URI as input entity -- Key: HIVE-8938 URL: https://issues.apache.org/jira/browse/HIVE-8938 Project: Hive Issue Type: Bug Components: Parser, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8938.1.patch, HIVE-8938.2.patch Compiler should capture the transform URI as input entity. This would enable better auditing for using transforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220307#comment-14220307 ] Prasad Mujumdar commented on HIVE-8893: --- Documented the new properties on the wiki. Thanks [~szehon]! Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Labels: TODOC15 Fix For: 0.15.0 Attachments: HIVE-8893.3.patch, HIVE-8893.4.patch, HIVE-8893.5.patch, HIVE-8893.6.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220375#comment-14220375 ] Prasad Mujumdar commented on HIVE-8893: --- [~leftylev] Thanks for bring that up. We do NOT need to add these new udf.* properties to the restrict list. Thes are only read at the bootup time from the initial configuration. Changing those in a session won't make a difference. I will add a note in the doc to mention that. Thanks! Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.3.patch, HIVE-8893.4.patch, HIVE-8893.5.patch, HIVE-8893.6.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220377#comment-14220377 ] Prasad Mujumdar commented on HIVE-8893: --- Updated the doc to mention the behavior. Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.3.patch, HIVE-8893.4.patch, HIVE-8893.5.patch, HIVE-8893.6.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8893: -- Attachment: HIVE-8893.6.patch Updated patch that addressed review feedback. Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.3.patch, HIVE-8893.4.patch, HIVE-8893.5.patch, HIVE-8893.6.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8266) create function using resource statement compilation should include resource URI entity
[ https://issues.apache.org/jira/browse/HIVE-8266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218743#comment-14218743 ] Prasad Mujumdar commented on HIVE-8266: --- [~leftylev] That's correct, it's not changing any user experience. Doesn't need a doc change. Thanks! create function using resource statement compilation should include resource URI entity - Key: HIVE-8266 URL: https://issues.apache.org/jira/browse/HIVE-8266 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8266.2.patch, HIVE-8266.3.patch The compiler add function name and db name as write entities for create function using resource statement. We should also include the resource URI path in the write entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8893: -- Attachment: HIVE-8893.5.patch Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.3.patch, HIVE-8893.4.patch, HIVE-8893.5.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8893: -- Attachment: (was: HIVE-8893.2.patch) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.3.patch, HIVE-8893.4.patch, HIVE-8893.5.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216472#comment-14216472 ] Prasad Mujumdar commented on HIVE-8893: --- The failed test optimize_nullscan passes in my setup. Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.3.patch, HIVE-8893.4.patch, HIVE-8893.5.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8829) Upgrade to Thrift 0.9.2
[
https://issues.apache.org/jira/browse/HIVE-8829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasad Mujumdar updated HIVE-8829:
--
Attachment: HIVE-8829.1.patch
Upgrade to Thrift 0.9.2
---
Key: HIVE-8829
URL: https://issues.apache.org/jira/browse/HIVE-8829
Project: Hive
Issue Type: Improvement
Affects Versions: 0.15.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Labels: HiveServer2, metastore
Fix For: 0.15.0
Attachments: HIVE-8829.1.patch
Apache Thrift 0.9.2 was released recently
(https://thrift.apache.org/download). It has a fix for THRIFT-2660 which can
cause HS2 (tcp mode) and Metastore processes to go OOM on getting a
non-thrift request when they use SASL transport. The reason ([thrift
code|https://github.com/apache/thrift/blob/0.9.x/lib/java/src/org/apache/thrift/transport/TSaslTransport.java#L177]):
{code}
protected SaslResponse receiveSaslMessage() throws TTransportException {
underlyingTransport.readAll(messageHeader, 0, messageHeader.length);
byte statusByte = messageHeader[0];
byte[] payload = new byte[EncodingUtils.decodeBigEndian(messageHeader,
STATUS_BYTES)];
underlyingTransport.readAll(payload, 0, payload.length);
NegotiationStatus status = NegotiationStatus.byValue(statusByte);
if (status == null) {
sendAndThrowMessage(NegotiationStatus.ERROR, Invalid status +
statusByte);
} else if (status == NegotiationStatus.BAD || status ==
NegotiationStatus.ERROR) {
try {
String remoteMessage = new String(payload, UTF-8);
throw new TTransportException(Peer indicated failure: +
remoteMessage);
} catch (UnsupportedEncodingException e) {
throw new TTransportException(e);
}
}
{code}
Basically since there are no message format checks / size checks before
creating the byte array, on getting a non-SASL message this creates a huge
byte array from some garbage size.
For HS2, an attempt was made to fix it here: HIVE-6468, which never went in.
I think for 0.15.0 it's best to upgarde to Thrift 0.9.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8829) Upgrade to Thrift 0.9.2
[
https://issues.apache.org/jira/browse/HIVE-8829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasad Mujumdar updated HIVE-8829:
--
Status: Patch Available (was: Open)
Upgrade to Thrift 0.9.2
---
Key: HIVE-8829
URL: https://issues.apache.org/jira/browse/HIVE-8829
Project: Hive
Issue Type: Improvement
Affects Versions: 0.15.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Labels: HiveServer2, metastore
Fix For: 0.15.0
Attachments: HIVE-8829.1.patch
Apache Thrift 0.9.2 was released recently
(https://thrift.apache.org/download). It has a fix for THRIFT-2660 which can
cause HS2 (tcp mode) and Metastore processes to go OOM on getting a
non-thrift request when they use SASL transport. The reason ([thrift
code|https://github.com/apache/thrift/blob/0.9.x/lib/java/src/org/apache/thrift/transport/TSaslTransport.java#L177]):
{code}
protected SaslResponse receiveSaslMessage() throws TTransportException {
underlyingTransport.readAll(messageHeader, 0, messageHeader.length);
byte statusByte = messageHeader[0];
byte[] payload = new byte[EncodingUtils.decodeBigEndian(messageHeader,
STATUS_BYTES)];
underlyingTransport.readAll(payload, 0, payload.length);
NegotiationStatus status = NegotiationStatus.byValue(statusByte);
if (status == null) {
sendAndThrowMessage(NegotiationStatus.ERROR, Invalid status +
statusByte);
} else if (status == NegotiationStatus.BAD || status ==
NegotiationStatus.ERROR) {
try {
String remoteMessage = new String(payload, UTF-8);
throw new TTransportException(Peer indicated failure: +
remoteMessage);
} catch (UnsupportedEncodingException e) {
throw new TTransportException(e);
}
}
{code}
Basically since there are no message format checks / size checks before
creating the byte array, on getting a non-SASL message this creates a huge
byte array from some garbage size.
For HS2, an attempt was made to fix it here: HIVE-6468, which never went in.
I think for 0.15.0 it's best to upgarde to Thrift 0.9.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8829) Upgrade to Thrift 0.9.2
[
https://issues.apache.org/jira/browse/HIVE-8829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216640#comment-14216640
]
Prasad Mujumdar commented on HIVE-8829:
---
[~vgumashta] I didn't notice that the ticket is assigned to you. If you
already have a patch, please feel free to ignore this one.
Upgrade to Thrift 0.9.2
---
Key: HIVE-8829
URL: https://issues.apache.org/jira/browse/HIVE-8829
Project: Hive
Issue Type: Improvement
Affects Versions: 0.15.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Labels: HiveServer2, metastore
Fix For: 0.15.0
Attachments: HIVE-8829.1.patch
Apache Thrift 0.9.2 was released recently
(https://thrift.apache.org/download). It has a fix for THRIFT-2660 which can
cause HS2 (tcp mode) and Metastore processes to go OOM on getting a
non-thrift request when they use SASL transport. The reason ([thrift
code|https://github.com/apache/thrift/blob/0.9.x/lib/java/src/org/apache/thrift/transport/TSaslTransport.java#L177]):
{code}
protected SaslResponse receiveSaslMessage() throws TTransportException {
underlyingTransport.readAll(messageHeader, 0, messageHeader.length);
byte statusByte = messageHeader[0];
byte[] payload = new byte[EncodingUtils.decodeBigEndian(messageHeader,
STATUS_BYTES)];
underlyingTransport.readAll(payload, 0, payload.length);
NegotiationStatus status = NegotiationStatus.byValue(statusByte);
if (status == null) {
sendAndThrowMessage(NegotiationStatus.ERROR, Invalid status +
statusByte);
} else if (status == NegotiationStatus.BAD || status ==
NegotiationStatus.ERROR) {
try {
String remoteMessage = new String(payload, UTF-8);
throw new TTransportException(Peer indicated failure: +
remoteMessage);
} catch (UnsupportedEncodingException e) {
throw new TTransportException(e);
}
}
{code}
Basically since there are no message format checks / size checks before
creating the byte array, on getting a non-SASL message this creates a huge
byte array from some garbage size.
For HS2, an attempt was made to fix it here: HIVE-6468, which never went in.
I think for 0.15.0 it's best to upgarde to Thrift 0.9.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
[ https://issues.apache.org/jira/browse/HIVE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216731#comment-14216731 ] Prasad Mujumdar commented on HIVE-8611: --- [~leftylev] Updated the wiki for config change. Thanks! grant/revoke syntax should support additional objects for authorization plugins --- Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8611.1.patch, HIVE-8611.2.patch, HIVE-8611.2.patch, HIVE-8611.3.patch, HIVE-8611.4.patch The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216744#comment-14216744 ] Prasad Mujumdar commented on HIVE-8612: --- [~leftylev] Documented the new config property on the metastore admin page. Thanks! Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8612.1.patch, HIVE-8612.2.patch, HIVE-8612.3.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8893: -- Attachment: HIVE-8893.3.patch Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.2.patch, HIVE-8893.3.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8893: -- Attachment: HIVE-8893.4.patch Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.2.patch, HIVE-8893.3.patch, HIVE-8893.4.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
Prasad Mujumdar created HIVE-8893: - Summary: Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8893: -- Status: Patch Available (was: Open) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.2.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8893) Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
[ https://issues.apache.org/jira/browse/HIVE-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8893: -- Attachment: HIVE-8893.2.patch Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode --- Key: HIVE-8893 URL: https://issues.apache.org/jira/browse/HIVE-8893 Project: Hive Issue Type: Bug Components: Authorization, HiveServer2, SQL Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8893.2.patch The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials. We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
[ https://issues.apache.org/jira/browse/HIVE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206117#comment-14206117 ] Prasad Mujumdar commented on HIVE-8611: --- The new config should go in the [https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization#SQLStandardBasedHiveAuthorization-Configuration|Authorization#SQLStandardBasedHiveAuthorization-Configuration] I will make the changes shortly. We don't need to change syntax doc. The syntax change as part of this patch is still just a wrapper, that functionality is not enabled in the HiveAuth yet. grant/revoke syntax should support additional objects for authorization plugins --- Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8611.1.patch, HIVE-8611.2.patch, HIVE-8611.2.patch, HIVE-8611.3.patch, HIVE-8611.4.patch The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
[ https://issues.apache.org/jira/browse/HIVE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8611: -- Resolution: Fixed Fix Version/s: (was: 0.14.0) 0.15.0 Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks for the review [~brocknoland]! grant/revoke syntax should support additional objects for authorization plugins --- Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8611.1.patch, HIVE-8611.2.patch, HIVE-8611.2.patch, HIVE-8611.3.patch, HIVE-8611.4.patch The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205707#comment-14205707 ] Prasad Mujumdar commented on HIVE-8612: --- The test failed vector_mapjoin_reduce.q seems to be unrelated to the change. It fails with and without the patch. Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8612.1.patch, HIVE-8612.2.patch, HIVE-8612.3.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8612: -- Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks [~brocknoland] for the review! Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8612.1.patch, HIVE-8612.2.patch, HIVE-8612.3.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8612: -- Fix Version/s: (was: 0.14.0) Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8612.1.patch, HIVE-8612.2.patch, HIVE-8612.3.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206065#comment-14206065 ] Prasad Mujumdar commented on HIVE-8612: --- [~leftylev] will update the documentation on wiki. It's only committed to trunk, updated the fix version. Thanks! Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8612.1.patch, HIVE-8612.2.patch, HIVE-8612.3.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
[ https://issues.apache.org/jira/browse/HIVE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8611: -- Attachment: HIVE-8611.4.patch Update patch to fix test failure due to new error message grant/revoke syntax should support additional objects for authorization plugins --- Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 Attachments: HIVE-8611.1.patch, HIVE-8611.2.patch, HIVE-8611.2.patch, HIVE-8611.3.patch, HIVE-8611.4.patch The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
[ https://issues.apache.org/jira/browse/HIVE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8611: -- Attachment: HIVE-8611.3.patch grant/revoke syntax should support additional objects for authorization plugins --- Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 Attachments: HIVE-8611.1.patch, HIVE-8611.2.patch, HIVE-8611.2.patch, HIVE-8611.3.patch The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8612: -- Attachment: HIVE-8612.3.patch Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8612.1.patch, HIVE-8612.2.patch, HIVE-8612.3.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8757) YARN dep in scheduler shim should be optional
[
https://issues.apache.org/jira/browse/HIVE-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199738#comment-14199738
]
Prasad Mujumdar commented on HIVE-8757:
---
+1 pending tests
[~brocknoland] Thanks for catching the problem and the fix!
YARN dep in scheduler shim should be optional
-
Key: HIVE-8757
URL: https://issues.apache.org/jira/browse/HIVE-8757
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-8757.patch
The {{hadoop-yarn-server-resourcemanager}} dep in the scheduler shim should
be optional so that yarn doesn't pollute dependent classpaths. Users who want
to use this feature must provide the yarn classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8692) Se
Prasad Mujumdar created HIVE-8692: - Summary: Se Key: HIVE-8692 URL: https://issues.apache.org/jira/browse/HIVE-8692 Project: Hive Issue Type: Bug Reporter: Prasad Mujumdar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8693) Separate out fair scheduler dependency from hadoop 0.23 shim
Prasad Mujumdar created HIVE-8693: - Summary: Separate out fair scheduler dependency from hadoop 0.23 shim Key: HIVE-8693 URL: https://issues.apache.org/jira/browse/HIVE-8693 Project: Hive Issue Type: Bug Components: HiveServer2, Shims Affects Versions: 0.14.0, 0.15.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar As part of HIVE-8424 HiveServer2 uses Fair scheduler APIs to determine resource queue allocation for non-impersonation case. This adds a hard dependency of Yarn server jars for Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8693) Separate out fair scheduler dependency from hadoop 0.23 shim
[ https://issues.apache.org/jira/browse/HIVE-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8693: -- Attachment: HIVE-8693.1.patch Separate out fair scheduler dependency from hadoop 0.23 shim Key: HIVE-8693 URL: https://issues.apache.org/jira/browse/HIVE-8693 Project: Hive Issue Type: Bug Components: HiveServer2, Shims Affects Versions: 0.14.0, 0.15.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8693.1.patch As part of HIVE-8424 HiveServer2 uses Fair scheduler APIs to determine resource queue allocation for non-impersonation case. This adds a hard dependency of Yarn server jars for Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8693) Separate out fair scheduler dependency from hadoop 0.23 shim
[ https://issues.apache.org/jira/browse/HIVE-8693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8693: -- Status: Patch Available (was: Open) Separate out fair scheduler dependency from hadoop 0.23 shim Key: HIVE-8693 URL: https://issues.apache.org/jira/browse/HIVE-8693 Project: Hive Issue Type: Bug Components: HiveServer2, Shims Affects Versions: 0.14.0, 0.15.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8693.1.patch As part of HIVE-8424 HiveServer2 uses Fair scheduler APIs to determine resource queue allocation for non-impersonation case. This adds a hard dependency of Yarn server jars for Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8634) HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly
[ https://issues.apache.org/jira/browse/HIVE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8634: -- Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks [~brocknoland] for the review! HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly Key: HIVE-8634 URL: https://issues.apache.org/jira/browse/HIVE-8634 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8634.1.patch The fair scheduler queue refresh in HiveServer2 (for non-impersonation mode), doesn't handle the primary/secondary queue mappings correctly. It's not reading primary and secondary rules from the scheduler rule file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8634) HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly
Prasad Mujumdar created HIVE-8634: - Summary: HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly Key: HIVE-8634 URL: https://issues.apache.org/jira/browse/HIVE-8634 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 The fair scheduler queue refresh in HiveServer2 (for non-impersonation mode), doesn't handle the primary/secondary queue mappings correctly. It's not reading primary and secondary rules from the scheduler rule file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8634) HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly
[ https://issues.apache.org/jira/browse/HIVE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8634: -- Status: Patch Available (was: Open) HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly Key: HIVE-8634 URL: https://issues.apache.org/jira/browse/HIVE-8634 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8634.1.patch The fair scheduler queue refresh in HiveServer2 (for non-impersonation mode), doesn't handle the primary/secondary queue mappings correctly. It's not reading primary and secondary rules from the scheduler rule file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8634) HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly
[ https://issues.apache.org/jira/browse/HIVE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8634: -- Attachment: HIVE-8634.1.patch HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly Key: HIVE-8634 URL: https://issues.apache.org/jira/browse/HIVE-8634 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8634.1.patch The fair scheduler queue refresh in HiveServer2 (for non-impersonation mode), doesn't handle the primary/secondary queue mappings correctly. It's not reading primary and secondary rules from the scheduler rule file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
Prasad Mujumdar created HIVE-8611: - Summary: grant/revoke syntax should support additional objects for authorization plugins Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
[ https://issues.apache.org/jira/browse/HIVE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8611: -- Attachment: HIVE-8611.1.patch grant/revoke syntax should support additional objects for authorization plugins --- Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 Attachments: HIVE-8611.1.patch The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8612) Support metadata result filter hooks
Prasad Mujumdar created HIVE-8612: - Summary: Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8612: -- Status: Patch Available (was: Open) Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8612.1.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8612: -- Attachment: HIVE-8612.1.patch Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8612.1.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
[ https://issues.apache.org/jira/browse/HIVE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8611: -- Status: Patch Available (was: Open) grant/revoke syntax should support additional objects for authorization plugins --- Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 Attachments: HIVE-8611.1.patch The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8611) grant/revoke syntax should support additional objects for authorization plugins
[ https://issues.apache.org/jira/browse/HIVE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8611: -- Attachment: HIVE-8611.2.patch Rebase with latest grant/revoke syntax should support additional objects for authorization plugins --- Key: HIVE-8611 URL: https://issues.apache.org/jira/browse/HIVE-8611 Project: Hive Issue Type: Bug Components: Authentication, SQL Affects Versions: 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 Attachments: HIVE-8611.1.patch, HIVE-8611.2.patch The authorization framework supports URI and global objects. The SQL syntax however doesn't allow granting privileges on these objects. We should allow the compiler to parse these so that it can be handled by authorization plugins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8612) Support metadata result filter hooks
[ https://issues.apache.org/jira/browse/HIVE-8612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8612: -- Attachment: HIVE-8612.2.patch Rebased with latest Support metadata result filter hooks Key: HIVE-8612 URL: https://issues.apache.org/jira/browse/HIVE-8612 Project: Hive Issue Type: Bug Components: Authorization, Metastore Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0, 0.15.0 Attachments: HIVE-8612.1.patch, HIVE-8612.2.patch Support metadata filter hook for metastore client. This will be useful for authorization plugins on hiveserver2 to filter metadata results, especially in case of non-impersonation mode where the metastore doesn't know the end user's identity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6715) Hive JDBC should include username into open session request for non-sasl connection
[ https://issues.apache.org/jira/browse/HIVE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173144#comment-14173144 ] Prasad Mujumdar commented on HIVE-6715: --- Thanks [~thejas]! Hive JDBC should include username into open session request for non-sasl connection --- Key: HIVE-6715 URL: https://issues.apache.org/jira/browse/HIVE-6715 Project: Hive Issue Type: Bug Components: JDBC Reporter: Srinath Assignee: Prasad Mujumdar Priority: Critical Fix For: 0.14.0 Attachments: HIVE-6715.1.patch, HIVE-6715.2.patch, HIVE-6715.3.patch The only parameter from sessVars that's being set in HiveConnection.openSession() is HS2_PROXY_USER. HIVE_AUTH_USER must also be set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8424) Support fair scheduler user queue mapping in non-impersonation mode
[ https://issues.apache.org/jira/browse/HIVE-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169766#comment-14169766 ] Prasad Mujumdar commented on HIVE-8424: --- That's correct. If the job authorization is enabled, then user hive needs have ALCs to access user queues. Support fair scheduler user queue mapping in non-impersonation mode --- Key: HIVE-8424 URL: https://issues.apache.org/jira/browse/HIVE-8424 Project: Hive Issue Type: Improvement Components: Shims Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8424.1.patch, HIVE-8424.patch Under non-impersonation mode, all MR jobs run as the hive system user. The default scheduler queue mapping is one queue per user. This is problematic for users who use the queues to regulate and track their MR resource usage. Yarn exposes an API to retrieve the fair scheduler queue mapping, which we can use to set the appropriate MR queue for the current user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8424) Support fair scheduler user queue mapping in non-impersonation mode
[ https://issues.apache.org/jira/browse/HIVE-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168867#comment-14168867 ] Prasad Mujumdar commented on HIVE-8424: --- To add to Szehon's comment, this patch is addressing the fair scheduler queue mapping for HiveServer2 running in non-impersonation mode. HiveServer2 can internally handle other processing, eg authorization correctly since it has the identity of the end user. However when at MR/Yarn level, all the the jobs are submitted as user hive. Hence the fair scheduler will use user Hive's queue for all Hive jobs. To address this issue, Yarn has exposed the queue mapping API for downstream services like Hive. This patch invokes that API to figure out the correct queue mapping from Yarn. Then explicitly sets that queue in the job configuration. This ensures that the jobs for the given use will get mapped to the appropriate queue. The patch is enabling this queue re-mapping in by default. This can be disabled by setting that config property to false. Support fair scheduler user queue mapping in non-impersonation mode --- Key: HIVE-8424 URL: https://issues.apache.org/jira/browse/HIVE-8424 Project: Hive Issue Type: Improvement Components: Shims Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8424.1.patch, HIVE-8424.patch Under non-impersonation mode, all MR jobs run as the hive system user. The default scheduler queue mapping is one queue per user. This is problematic for users who use the queues to regulate and track their MR resource usage. Yarn exposes an API to retrieve the fair scheduler queue mapping, which we can use to set the appropriate MR queue for the current user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8424) Support fair scheduler user queue mapping in non-impersonation mode
[ https://issues.apache.org/jira/browse/HIVE-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168869#comment-14168869 ] Prasad Mujumdar commented on HIVE-8424: --- [~mohitsabharwal] Changes look fine to me. Please take a look at Szehon's suggesting regarding the pom changes. Thanks! Support fair scheduler user queue mapping in non-impersonation mode --- Key: HIVE-8424 URL: https://issues.apache.org/jira/browse/HIVE-8424 Project: Hive Issue Type: Improvement Components: Shims Reporter: Mohit Sabharwal Assignee: Mohit Sabharwal Attachments: HIVE-8424.1.patch, HIVE-8424.patch Under non-impersonation mode, all MR jobs run as the hive system user. The default scheduler queue mapping is one queue per user. This is problematic for users who use the queues to regulate and track their MR resource usage. Yarn exposes an API to retrieve the fair scheduler queue mapping, which we can use to set the appropriate MR queue for the current user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166473#comment-14166473
]
Prasad Mujumdar commented on HIVE-7932:
---
Patch backported to 0.14. Thanks Vikram!
It may cause NP exception when add accessed columns to ReadEntity
-
Key: HIVE-7932
URL: https://issues.apache.org/jira/browse/HIVE-7932
Project: Hive
Issue Type: Bug
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Fix For: 0.15.0
Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch
{code}
case TABLE:
entity.getAccessedColumns().addAll(
tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
{code}
if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null,
addAll(null) will throw null pointer exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164320#comment-14164320
]
Prasad Mujumdar commented on HIVE-7932:
---
+1
Looks fine to me. Thanks for adding the testcase.
It may cause NP exception when add accessed columns to ReadEntity
-
Key: HIVE-7932
URL: https://issues.apache.org/jira/browse/HIVE-7932
Project: Hive
Issue Type: Bug
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch
{code}
case TABLE:
entity.getAccessedColumns().addAll(
tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
{code}
if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null,
addAll(null) will throw null pointer exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasad Mujumdar updated HIVE-7932:
--
Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)
Patch committed to trunk. Thanks [~Huang Xiaomeng]!
It may cause NP exception when add accessed columns to ReadEntity
-
Key: HIVE-7932
URL: https://issues.apache.org/jira/browse/HIVE-7932
Project: Hive
Issue Type: Bug
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Fix For: 0.15.0
Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch
{code}
case TABLE:
entity.getAccessedColumns().addAll(
tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
{code}
if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null,
addAll(null) will throw null pointer exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164534#comment-14164534 ] Prasad Mujumdar commented on HIVE-8083: --- [~leftylev] Thanks for pointing that out. Just updated the wiki. Authorization DDLs should not enforce hive identifier syntax for user or group -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.0, 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch, HIVE-8083.3.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8083: -- Release Note: The user name in the grant and revoke statements may be optionally surrounded by backtick characters (`) irrespective of the hive.support.quoted.identifiers setting. Authorization DDLs should not enforce hive identifier syntax for user or group -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.0, 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch, HIVE-8083.3.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164544#comment-14164544
]
Prasad Mujumdar commented on HIVE-7932:
---
[~vikram.dixit] Requesting backport to 0.14. It's a followup to HIVE-7730 which
is already in 0.14. Thanks!
It may cause NP exception when add accessed columns to ReadEntity
-
Key: HIVE-7932
URL: https://issues.apache.org/jira/browse/HIVE-7932
Project: Hive
Issue Type: Bug
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Fix For: 0.15.0
Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch
{code}
case TABLE:
entity.getAccessedColumns().addAll(
tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
{code}
if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null,
addAll(null) will throw null pointer exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8266) create function using resource statement compilation should include resource URI entity
[ https://issues.apache.org/jira/browse/HIVE-8266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8266: -- Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks [~brocknoland] for the review! create function using resource statement compilation should include resource URI entity - Key: HIVE-8266 URL: https://issues.apache.org/jira/browse/HIVE-8266 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.15.0 Attachments: HIVE-8266.2.patch, HIVE-8266.3.patch The compiler add function name and db name as write entities for create function using resource statement. We should also include the resource URI path in the write entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8266) create function using resource statement compilation should include resource URI entity
[ https://issues.apache.org/jira/browse/HIVE-8266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8266: -- Attachment: HIVE-8266.3.patch Updated the test output of the failed test to add the new write entity printed by the test hook. create function using resource statement compilation should include resource URI entity - Key: HIVE-8266 URL: https://issues.apache.org/jira/browse/HIVE-8266 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8266.2.patch, HIVE-8266.3.patch The compiler add function name and db name as write entities for create function using resource statement. We should also include the resource URI path in the write entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8266) create function using resource statement compilation should include resource URI entity
[ https://issues.apache.org/jira/browse/HIVE-8266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8266: -- Attachment: HIVE-8266.2.patch create function using resource statement compilation should include resource URI entity - Key: HIVE-8266 URL: https://issues.apache.org/jira/browse/HIVE-8266 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8266.2.patch The compiler add function name and db name as write entities for create function using resource statement. We should also include the resource URI path in the write entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8266) create function using resource statement compilation should include resource URI entity
[ https://issues.apache.org/jira/browse/HIVE-8266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8266: -- Status: Patch Available (was: Open) create function using resource statement compilation should include resource URI entity - Key: HIVE-8266 URL: https://issues.apache.org/jira/browse/HIVE-8266 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8266.2.patch The compiler add function name and db name as write entities for create function using resource statement. We should also include the resource URI path in the write entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8266) create function using resource statement compilation should include resource URI entity
Prasad Mujumdar created HIVE-8266: - Summary: create function using resource statement compilation should include resource URI entity Key: HIVE-8266 URL: https://issues.apache.org/jira/browse/HIVE-8266 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar The compiler add function name and db name as write entities for create function using resource statement. We should also include the resource URI path in the write entity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139251#comment-14139251 ] Prasad Mujumdar commented on HIVE-8083: --- The failure look unrelated. This test TestHCatLoader#testReadDataPrimitiveTypes passes in my setup with this patch. Authorization DDLs should not enforce hive identifier syntax for user or group -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.0, 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch, HIVE-8083.3.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8083: -- Attachment: HIVE-8083.3.patch Updated patch. Addresses review feedback. Authorization DDLs should not enforce hive identifier syntax for user or group -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.0, 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch, HIVE-8083.3.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8139) Upgrade commons-lang from 2.4 to 2.6
Prasad Mujumdar created HIVE-8139: - Summary: Upgrade commons-lang from 2.4 to 2.6 Key: HIVE-8139 URL: https://issues.apache.org/jira/browse/HIVE-8139 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 Upgrade commons-lang version from 2.4 to latest 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8139) Upgrade commons-lang from 2.4 to 2.6
[ https://issues.apache.org/jira/browse/HIVE-8139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8139: -- Attachment: HIVE-8139.1.patch Upgrade commons-lang from 2.4 to 2.6 Key: HIVE-8139 URL: https://issues.apache.org/jira/browse/HIVE-8139 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 Attachments: HIVE-8139.1.patch Upgrade commons-lang version from 2.4 to latest 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8139) Upgrade commons-lang from 2.4 to 2.6
[ https://issues.apache.org/jira/browse/HIVE-8139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8139: -- Status: Patch Available (was: Open) Upgrade commons-lang from 2.4 to 2.6 Key: HIVE-8139 URL: https://issues.apache.org/jira/browse/HIVE-8139 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.14.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.14.0 Attachments: HIVE-8139.1.patch Upgrade commons-lang version from 2.4 to latest 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8083: -- Attachment: HIVE-8083.2.patch Rebased with latest Authorization DDLs should not enforce hive identifier syntax for user or group -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.0, 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132811#comment-14132811 ] Prasad Mujumdar commented on HIVE-8083: --- The test failures don't look related to the patch. Authorization DDLs should not enforce hive identifier syntax for user or group -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.0, 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8083.1.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132828#comment-14132828
]
Prasad Mujumdar commented on HIVE-7932:
---
[~Huang Xiaomeng] Thanks for finding the issue and providing a patch.
The change look fine to me. Would it be possible to add a simple testcase that
verifies this codepath ?
It may cause NP exception when add accessed columns to ReadEntity
-
Key: HIVE-7932
URL: https://issues.apache.org/jira/browse/HIVE-7932
Project: Hive
Issue Type: Bug
Reporter: Xiaomeng Huang
Assignee: Xiaomeng Huang
Attachments: HIVE-7932.001.patch
{code}
case TABLE:
entity.getAccessedColumns().addAll(
tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
{code}
if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null,
addAll(null) will throw null pointer exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group namesname that
Prasad Mujumdar created HIVE-8083: - Summary: Authorization DDLs should not enforce hive identifier syntax for user or group namesname that Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.1, 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group namesname that
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8083: -- Status: Patch Available (was: Open) Authorization DDLs should not enforce hive identifier syntax for user or group namesname that -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.1, 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8083.1.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group namesname that
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8083: -- Attachment: HIVE-8083.1.patch Authorization DDLs should not enforce hive identifier syntax for user or group namesname that -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.0, 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8083.1.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8083: -- Summary: Authorization DDLs should not enforce hive identifier syntax for user or group (was: Authorization DDLs should not enforce hive identifier syntax for user or group namesname that ) Authorization DDLs should not enforce hive identifier syntax for user or group -- Key: HIVE-8083 URL: https://issues.apache.org/jira/browse/HIVE-8083 Project: Hive Issue Type: Bug Components: SQL, SQLStandardAuthorization Affects Versions: 0.13.0, 0.13.1 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-8083.1.patch The compiler expects principals (user, group and role) as hive identifiers for authorization DDLs. The user and group are entities that belong to external namespace and we can't expect those to follow hive identifier syntax rules. For example, a userid or group can contain '-' which is not allowed by compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7890) SessionState creates HMS Client while not impersonating
[
https://issues.apache.org/jira/browse/HIVE-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118350#comment-14118350
]
Prasad Mujumdar commented on HIVE-7890:
---
Looks fine to me.
HS2 handles the thrift connection to HMS by recreating a new connection for new
session in case of impersonation. The HiveDB object was not handled in the same
way. This patch is handling that more efficiently.
+1
SessionState creates HMS Client while not impersonating
---
Key: HIVE-7890
URL: https://issues.apache.org/jira/browse/HIVE-7890
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Brock Noland
Attachments: HIVE-7890.2.patch
In SessionState.start [an instance of the the HMSClient is
created|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L367].
When impersonation is enabled, this call does not occur within a doas call
and thus the HMSClient is created as the server user, not the impersonated
user.
Thus calls to the HMS are made by the hive user as opposed to the end user.
This causes file ownership such as a database directory owner to be
incorrect. While debugging this, I got stack trace below. As you can see we
are calling getMSC without a doas.
{noformat}
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2474)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:367)
at
org.apache.hive.service.cli.session.HiveSessionImpl.init(HiveSessionImpl.java:121)
at
org.apache.hive.service.cli.session.HiveSessionImplwithUGI.init(HiveSessionImplwithUGI.java:49)
at
org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:130)
at
org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:163)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:290)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:208)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7682) HadoopThriftAuthBridge20S should not reset configuration unless required
[ https://issues.apache.org/jira/browse/HIVE-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118556#comment-14118556 ] Prasad Mujumdar commented on HIVE-7682: --- [~brocknoland] Looks fine to me. A couple of minor comments on RB. Thanks! HadoopThriftAuthBridge20S should not reset configuration unless required Key: HIVE-7682 URL: https://issues.apache.org/jira/browse/HIVE-7682 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-7682.1.patch, HIVE-7682.2.patch In HadoopThriftAuthBridge20S methods createClientWithConf and getCurrentUGIWithConf we create new Configuration objects so we can set the authentication type. When loading the new Configuration object, it looks like core-site.xml for the cluster it's connected to. This causes issues for Oozie since oozie does not have access to the core-site.xml as it's cluster agnostic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7682) HadoopThriftAuthBridge20S should not reset configuration unless required
[ https://issues.apache.org/jira/browse/HIVE-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118677#comment-14118677 ] Prasad Mujumdar commented on HIVE-7682: --- Patch 3 Looks fine to me. +1 HadoopThriftAuthBridge20S should not reset configuration unless required Key: HIVE-7682 URL: https://issues.apache.org/jira/browse/HIVE-7682 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-7682.1.patch, HIVE-7682.2.patch, HIVE-7682.3.patch In HadoopThriftAuthBridge20S methods createClientWithConf and getCurrentUGIWithConf we create new Configuration objects so we can set the authentication type. When loading the new Configuration object, it looks like core-site.xml for the cluster it's connected to. This causes issues for Oozie since oozie does not have access to the core-site.xml as it's cluster agnostic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7342) support hiveserver2,metastore specific config files
[ https://issues.apache.org/jira/browse/HIVE-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063969#comment-14063969 ] Prasad Mujumdar commented on HIVE-7342: --- [~thejas] The patch look fine to me. Just wondering if it would make sense to further split the metastore config into client (or base) and server. There are common configs like setugi, enableSasl etc that need to be in sync on both client and server. If those are available in a common file, it will be less prone to incompatible configs. The server will load both base and server specific configs, the client will only load the base config. support hiveserver2,metastore specific config files --- Key: HIVE-7342 URL: https://issues.apache.org/jira/browse/HIVE-7342 Project: Hive Issue Type: Bug Components: Configuration, HiveServer2, Metastore Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-7342.1.patch, HIVE-7342.2.patch There is currently a single configuration file for all components in hive. ie, components such as hive cli, hiveserver2 and metastore all read from the same hive-site.xml. It will be useful to have a server specific hive-site.xml, so that you can have some different configuration value set for a server. For example, you might want to enabled authorization checks for hiveserver2, while disabling the checks for hive cli. The workaround today is to add any component specific configuration as a commandline (-hiveconf) argument. Using server specific config files (eg hiveserver2-site.xml, hivemetastore-site.xml) that override the entries in hive-site.xml will make the configuration much more easy to manage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7342) support hiveserver2,metastore specific config files
[ https://issues.apache.org/jira/browse/HIVE-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064062#comment-14064062 ] Prasad Mujumdar commented on HIVE-7342: --- I guess treating hive-site as base file should be sufficient. It's unlikely that you will have variety of metastore setups (embedded and remote, or secure and unsecure) in a single deployment. Thanks for updating the release notes! +1 support hiveserver2,metastore specific config files --- Key: HIVE-7342 URL: https://issues.apache.org/jira/browse/HIVE-7342 Project: Hive Issue Type: Bug Components: Configuration, HiveServer2, Metastore Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-7342.1.patch, HIVE-7342.2.patch There is currently a single configuration file for all components in hive. ie, components such as hive cli, hiveserver2 and metastore all read from the same hive-site.xml. It will be useful to have a server specific hive-site.xml, so that you can have some different configuration value set for a server. For example, you might want to enabled authorization checks for hiveserver2, while disabling the checks for hive cli. The workaround today is to add any component specific configuration as a commandline (-hiveconf) argument. Using server specific config files (eg hiveserver2-site.xml, hivemetastore-site.xml) that override the entries in hive-site.xml will make the configuration much more easy to manage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6891) Alter rename partition Perm inheritance and general partition/table group inheritance
[ https://issues.apache.org/jira/browse/HIVE-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-6891: -- Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks [~szehon] for the contribution, thanks [~brocknoland] for the review. Alter rename partition Perm inheritance and general partition/table group inheritance - Key: HIVE-6891 URL: https://issues.apache.org/jira/browse/HIVE-6891 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Fix For: 0.14.0 Attachments: HIVE-6891.2.patch, HIVE-6891.3.patch, HIVE-6891.4.patch, HIVE-6891.patch Found this issue while looking at the method mentioned by HIVE-6648. 'alter table .. partition .. rename to ..' and other commands calling Warehouse.mkdirs() doesn't inherit permission on the partition directories and consequently the data, when hive.warehouse.subdir.inherit.perms is set. Also, in these scenarios of directory creation, group is not being inherited. Data files are already inheriting group by HIVE-3756. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6800) HiveServer2 is not passing proxy user setting through hive-site
[
https://issues.apache.org/jira/browse/HIVE-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13956519#comment-13956519
]
Prasad Mujumdar commented on HIVE-6800:
---
[~vaibhavgumashta] Thanks for fixing the issue. Looks fine to me.
+1
HiveServer2 is not passing proxy user setting through hive-site
---
Key: HIVE-6800
URL: https://issues.apache.org/jira/browse/HIVE-6800
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Fix For: 0.13.0
Attachments: HIVE-6800.1.patch
Setting the following in core-site.xml works fine in a secure cluster with
hive.server2.allow.user.substitution set to true:
{code}
property
namehadoop.proxyuser.user1.groups/name
valueusers/value
/property
property
namehadoop.proxyuser.user1.hosts/name
value*/value
/property
{code}
where user1 will be proxying for user2:
{code}
!connect
jdbc:hive2:/myhostname:1/;principal=hive/_h...@example.com;hive.server2.proxy.user=user2
user1 fakepwd org.apache.hive.jdbc.HiveDriver
{code}
However, setting this in hive-site.xml throws Failed to validate proxy
privilage exception.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-2539) Enable passing username/password via JDBC
[ https://issues.apache.org/jira/browse/HIVE-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13948455#comment-13948455 ] Prasad Mujumdar commented on HIVE-2539: --- [~qwertymaniac] Thanks for bringing that up. HiveServer2 and it's JDBC driver support multiple authentication mechanisms. I will go ahead and close the issue. [~mac.had...@gmail.com] and [~xcqnoah] Just in case, if you are still interested in submitting the patch for old HiveServer, please feel free to reopen the ticket. Thanks! Enable passing username/password via JDBC - Key: HIVE-2539 URL: https://issues.apache.org/jira/browse/HIVE-2539 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.7.1 Reporter: Sriram Krishnan Assignee: chunqing xie Labels: patch Attachments: HIVE-2539.PATCH Changing the username and/or the password seems to have no effect (also confirmed here: https://cwiki.apache.org/Hive/hivejdbcinterface.html). Connection con = DriverManager.getConnection(jdbc:hive://localhost:1/default, , ); Would be beneficial to pass the username/password via JDBC - and also for the server to honor the username password being passed (may be dependent of that being fixed first). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HIVE-2539) Enable passing username/password via JDBC
[ https://issues.apache.org/jira/browse/HIVE-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar resolved HIVE-2539. --- Resolution: Implemented The JDBC driver for HiveServer2 support user/password authentication via JDBC getConnection() API. Enable passing username/password via JDBC - Key: HIVE-2539 URL: https://issues.apache.org/jira/browse/HIVE-2539 Project: Hive Issue Type: Improvement Components: JDBC Affects Versions: 0.7.1 Reporter: Sriram Krishnan Assignee: chunqing xie Labels: patch Attachments: HIVE-2539.PATCH Changing the username and/or the password seems to have no effect (also confirmed here: https://cwiki.apache.org/Hive/hivejdbcinterface.html). Connection con = DriverManager.getConnection(jdbc:hive://localhost:1/default, , ); Would be beneficial to pass the username/password via JDBC - and also for the server to honor the username password being passed (may be dependent of that being fixed first). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6741) HiveServer2 startup fails in secure (kerberos) mode due to backward incompatible hadoop change
[
https://issues.apache.org/jira/browse/HIVE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946756#comment-13946756
]
Prasad Mujumdar commented on HIVE-6741:
---
[~vaibhavgumashta] Thanks for looking into the problem.
[HADOOP-10211|https://issues.apache.org/jira/browse/HADOOP-10211] seems to be
fixed in Hadoop 2.4 and trunk (3.0), which are both unreleased hadoop versions.
Hive is still using last hadoop release 2.3.0. Shouldn't this wait till we
upgrade hadoop dependencies ?
HiveServer2 startup fails in secure (kerberos) mode due to backward
incompatible hadoop change
--
Key: HIVE-6741
URL: https://issues.apache.org/jira/browse/HIVE-6741
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Blocker
Fix For: 0.13.0
Attachments: HIVE-6741.1.patch
[HADOOP-10211|https://issues.apache.org/jira/browse/HADOOP-10211] made a
backward incompatible change due to which the following hive call returns a
null map
([HiveAuthFactory-old|https://github.com/apache/hive/blob/fc3fdb19668369c56994d11df3207e14f2c5dba8/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java#L115]):
{code}
MapString, String hadoopSaslProps = ShimLoader.getHadoopThriftAuthBridge().
getHadoopSaslProperties(conf);
SaslQOP hadoopSaslQOP = SaslQOP.fromString(hadoopSaslProps.get(Sasl.QOP));
if(hadoopSaslQOP.ordinal() saslQOP.ordinal()) {
LOG.warn(MessageFormat.format(\hadoop.rpc.protection\ is set to higher
security level +
{0} then {1} which is set to {2}, hadoopSaslQOP.toString(),
ConfVars.HIVE_SERVER2_THRIFT_SASL_QOP.varname, saslQOP.toString()));
}
{code}
Since this code path is only used for logging hadoop sasl qop values in case
hadoop's qop hive's qop, we can do away with this and add a general log
message.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6741) HiveServer2 startup fails in secure (kerberos) mode due to backward incompatible hadoop change
[
https://issues.apache.org/jira/browse/HIVE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947145#comment-13947145
]
Prasad Mujumdar commented on HIVE-6741:
---
[~vaibhavgumashta] This is an incompatible change and hence would be better to
do this along with the hadoop upgrade. The code being removed is applicable to
Hadoop 2.3. We might not switch to Hadoop 2.4 at all for Hive 0.14 as well, or
may be Hadoop 2.4 will change the logic later during it's release cycle etc. In
such cases the patch won't be adding any value.
Also [https://issues.apache.org/jira/browse/HIVE-6657|HIVE-6657] is proposing
to add miniKdc based test in Hive. I guess that would catch this problem once
we upgrade, so won't miss out this patch.
Regarding the actual patch, is the new unconditional log message needed at all
? It's getting printed be printed regardless the hadoop.rpc.protection and
hive.server2.thrift.sasl.qop config. Would it make sense to just add it to docs
and remove the message ?
HiveServer2 startup fails in secure (kerberos) mode due to backward
incompatible hadoop change
--
Key: HIVE-6741
URL: https://issues.apache.org/jira/browse/HIVE-6741
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.14.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Blocker
Fix For: 0.14.0
Attachments: HIVE-6741.1.patch
[HADOOP-10211|https://issues.apache.org/jira/browse/HADOOP-10211] made a
backward incompatible change due to which the following hive call returns a
null map
([HiveAuthFactory-old|https://github.com/apache/hive/blob/fc3fdb19668369c56994d11df3207e14f2c5dba8/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java#L115]):
{code}
MapString, String hadoopSaslProps = ShimLoader.getHadoopThriftAuthBridge().
getHadoopSaslProperties(conf);
SaslQOP hadoopSaslQOP = SaslQOP.fromString(hadoopSaslProps.get(Sasl.QOP));
if(hadoopSaslQOP.ordinal() saslQOP.ordinal()) {
LOG.warn(MessageFormat.format(\hadoop.rpc.protection\ is set to higher
security level +
{0} then {1} which is set to {2}, hadoopSaslQOP.toString(),
ConfVars.HIVE_SERVER2_THRIFT_SASL_QOP.varname, saslQOP.toString()));
}
{code}
Since this code path is only used for logging hadoop sasl qop values in case
hadoop's qop hive's qop, we can do away with this and add a general log
message.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-6746) Hive should be able to retrieve logs for failed MR2 jobs
Prasad Mujumdar created HIVE-6746: - Summary: Hive should be able to retrieve logs for failed MR2 jobs Key: HIVE-6746 URL: https://issues.apache.org/jira/browse/HIVE-6746 Project: Hive Issue Type: Improvement Components: Diagnosability, Logging, Shims Affects Versions: 0.12.0, 0.13.0 Reporter: Prasad Mujumdar The Hadoop 0.23 shim doesn't support retrieving Yarn logs for failed tasks. This should be supported in order to improve debugging/troubleshooting of query failures. -- This message was sent by Atlassian JIRA (v6.2#6252)
