[jira] [Commented] (HIVE-1537) Allow users to specify LOCATION in CREATE DATABASE statement
[ https://issues.apache.org/jira/browse/HIVE-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13059176#comment-13059176 ] Devaraj Das commented on HIVE-1537: --- I'm no longer at Yahoo!. My new email address is d...@hortonworks.com. Please update your address book and resend your message. Allow users to specify LOCATION in CREATE DATABASE statement Key: HIVE-1537 URL: https://issues.apache.org/jira/browse/HIVE-1537 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Carl Steinbach Assignee: Thiruvel Thirumoolan Attachments: HIVE-1537.patch, hive-1537.metastore.part.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1537) Allow users to specify LOCATION in CREATE DATABASE statement
[ https://issues.apache.org/jira/browse/HIVE-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1537: -- Attachment: hive-1537.metastore.part.patch Assuming that the location issue is solved, we need some checks in the create_database handler similar to what is there in the create_table handler. This patch is an example patch. It introduces a check in HiveMetaStore.create_database_core for existence of database directory, and also checks for failure to create one. In either case, the create_database_core operation throws an exception and the DDL would fail. Allow users to specify LOCATION in CREATE DATABASE statement Key: HIVE-1537 URL: https://issues.apache.org/jira/browse/HIVE-1537 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Carl Steinbach Assignee: Thiruvel Thirumoolan Attachments: hive-1537.metastore.part.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1943) Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data
[ https://issues.apache.org/jira/browse/HIVE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1943: -- Fix Version/s: 0.8.0 Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data - Key: HIVE-1943 URL: https://issues.apache.org/jira/browse/HIVE-1943 Project: Hive Issue Type: Bug Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.8.0 Attachments: hive-metastore-auth.patch Currently, metastore operations with associated hdfs operations like drop_partition doesn't do a rollback if the hdfs operation fails. The metastore first updates the metadata, and then tries to do hdfs operation. If the hdfs operation fails for any reason, the data on the hdfs will be orphaned. We should improve the situation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1943) Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data
[ https://issues.apache.org/jira/browse/HIVE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1943: -- Attachment: hive-metastore-auth.patch Attaching the patch that I uploaded last on reviewboard. Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data - Key: HIVE-1943 URL: https://issues.apache.org/jira/browse/HIVE-1943 Project: Hive Issue Type: Bug Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.8.0 Attachments: hive-metastore-auth.patch Currently, metastore operations with associated hdfs operations like drop_partition doesn't do a rollback if the hdfs operation fails. The metastore first updates the metadata, and then tries to do hdfs operation. If the hdfs operation fails for any reason, the data on the hdfs will be orphaned. We should improve the situation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1943) Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data
[ https://issues.apache.org/jira/browse/HIVE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1943: -- Assignee: Devaraj Das Summary: Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data (was: Metastore operations should do a 'rollback' for HDFS failures) Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data - Key: HIVE-1943 URL: https://issues.apache.org/jira/browse/HIVE-1943 Project: Hive Issue Type: Bug Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Currently, metastore operations with associated hdfs operations like drop_partition doesn't do a rollback if the hdfs operation fails. The metastore first updates the metadata, and then tries to do hdfs operation. If the hdfs operation fails for any reason, the data on the hdfs will be orphaned. We should improve the situation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1943) Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data
[ https://issues.apache.org/jira/browse/HIVE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13019083#comment-13019083 ] Devaraj Das commented on HIVE-1943: --- Created a reviewboard review request - https://reviews.apache.org/r/586/ The patch basically adds checks before updating metadata during operations like drop_table, drop_database, etc. The checks are at the Warehouse layer to see whether the associated data can be really deleted by the user in question. If the data can't be deleted then the metadata operation is not done. This check is configurable and can be disabled. Metastore operations (like drop_partition) could be improved in terms of maintaining consistency of metadata and data - Key: HIVE-1943 URL: https://issues.apache.org/jira/browse/HIVE-1943 Project: Hive Issue Type: Bug Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Currently, metastore operations with associated hdfs operations like drop_partition doesn't do a rollback if the hdfs operation fails. The metastore first updates the metadata, and then tries to do hdfs operation. If the hdfs operation fails for any reason, the data on the hdfs will be orphaned. We should improve the situation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1988) Make the delegation token issued by the MetaStore owned by the right user
[ https://issues.apache.org/jira/browse/HIVE-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1988: -- Attachment: hive-1988-5.1.patch The same patch that i uploaded on reviewboard.. Make the delegation token issued by the MetaStore owned by the right user - Key: HIVE-1988 URL: https://issues.apache.org/jira/browse/HIVE-1988 Project: Hive Issue Type: Bug Components: Metastore, Security, Server Infrastructure Affects Versions: 0.7.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.8.0 Attachments: hive-1988-3.patch, hive-1988-5.1.patch, hive-1988.patch The 'owner' of any delegation token issued by the MetaStore is set to the requesting user. When a delegation token is asked by the user himself during a job submission, this is fine. However, in the case where the token is requested for by services (e.g., Oozie), on behalf of the user, the token's owner is set to the user the service is running as. Later on, when the token is used by a MapReduce task, the MetaStore treats the incoming request as coming from Oozie and does operations as Oozie. This means any new directory creations (e.g., create_table) on the hdfs by the MetaStore will end up with Oozie as the owner. Also, the MetaStore doesn't check whether a user asking for a token on behalf of some other user, is actually authorized to act on behalf of that other user. We should start using the ProxyUser authorization in the MetaStore (HADOOP-6510's APIs). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2079) The warehouse directory shouldn't be 777'ed
[ https://issues.apache.org/jira/browse/HIVE-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013557#comment-13013557 ] Devaraj Das commented on HIVE-2079: --- Edward, at Yahoo!, we run the thrift server as a standalone metastore server, and there the problem can be handled. The solution is still under investigation but here is the flow of the directory creations and permission settings: 1) Have the real warehouse directory owned by the hive-thrift-server user and let that have 755 permissions. 2) Have a temp warehouse directory for staging the creation of tables/databases, and let that have 777 permissions. 3) When a user issues a create_table/database command, the hive-thrift-server creates the corresponding directory in the temp location. This operation happens as the user in question and the directory ends up getting owned by the user. 4) The hive-thrift-server then moves the directory to the real warehouse directory. This operation is done as the hive-thrift-server user.Since the temp directory has 777 permissions, and the real warehouse directory is owned by the hive-thrift-server user, the move will succeed. With all the work that has been done in mostly HIVE-1842 HIVE-1696, the above seems possible. Granted, this won't work when hive runs in the fat-client mode. So, most likely, we will make the above be based on whether metastore is running in the local mode or not (hive.metastore.local config). Makes sense ? The warehouse directory shouldn't be 777'ed --- Key: HIVE-2079 URL: https://issues.apache.org/jira/browse/HIVE-2079 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Devaraj Das Assignee: Mac Yang Fix For: 0.8.0 The warehouse directory is created with a permissions of 777. This is to allow any user to successfully create database/table directories there. The security issue is that anyone can delete any directory in the warehouse. We should fix this hole. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-78) Authorization infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13013754#comment-13013754 ] Devaraj Das commented on HIVE-78: - BTW, was any thought put in to implement the authorization checks in the ObjectStore? In the model where a MetaStore server is deployed separately, applications (map/reduce tasks for example), can make programmatic calls to the MetaStore to, for example, drop random tables/partitions, and they will pass.. Just wondering whether this usecase was considered. Authorization infrastructure for Hive - Key: HIVE-78 URL: https://issues.apache.org/jira/browse/HIVE-78 Project: Hive Issue Type: New Feature Components: Metastore, Query Processor, Security, Server Infrastructure Reporter: Ashish Thusoo Assignee: He Yongqiang Fix For: 0.7.0 Attachments: HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, HIVE-78.10.no_thrift.patch, HIVE-78.11.patch, HIVE-78.12.2.patch, HIVE-78.12.3.patch, HIVE-78.12.4.patch, HIVE-78.12.5.patch, HIVE-78.12.patch, HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, HIVE-78.5.no_thrift.patch, HIVE-78.6.complete.patch, HIVE-78.6.no_thrift.patch, HIVE-78.7.no_thrift.patch, HIVE-78.7.patch, HIVE-78.9.no_thrift.patch, HIVE-78.9.patch, createuser-v1.patch, hive-78-metadata-v1.patch, hive-78-syntax-v1.patch, hive-78.diff Allow hive to integrate with existing user repositories for authentication and authorization infromation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1988) Make the delegation token issued by the MetaStore owned by the right user
[ https://issues.apache.org/jira/browse/HIVE-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13012279#comment-13012279 ] Devaraj Das commented on HIVE-1988: --- I updated the reviewboard https://reviews.apache.org/r/528/ with some changes. The main change is: The methods getDelegationToken/renewDelegationToken now check for the authentication method being KERBEROS. If not, then it refuses to give a delegation token. This takes care of a security hole, where, if a delegation token has been compromised, the malicious user in possession of the token could use it to authenticate itself with the metastore, and get a new delegation token. This process could go forever (and hence the malicious user could access the metastore without ever going through a kerberos authentication). Making the handing out of delegation tokens based on a prior kerberos authentication limits this. Also, the patch on reviewboard doesn't have generated code. I will upload it once someone takes a look at the patch and gives feedback. Make the delegation token issued by the MetaStore owned by the right user - Key: HIVE-1988 URL: https://issues.apache.org/jira/browse/HIVE-1988 Project: Hive Issue Type: Bug Components: Metastore, Security, Server Infrastructure Affects Versions: 0.7.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.8.0 Attachments: hive-1988-3.patch, hive-1988.patch The 'owner' of any delegation token issued by the MetaStore is set to the requesting user. When a delegation token is asked by the user himself during a job submission, this is fine. However, in the case where the token is requested for by services (e.g., Oozie), on behalf of the user, the token's owner is set to the user the service is running as. Later on, when the token is used by a MapReduce task, the MetaStore treats the incoming request as coming from Oozie and does operations as Oozie. This means any new directory creations (e.g., create_table) on the hdfs by the MetaStore will end up with Oozie as the owner. Also, the MetaStore doesn't check whether a user asking for a token on behalf of some other user, is actually authorized to act on behalf of that other user. We should start using the ProxyUser authorization in the MetaStore (HADOOP-6510's APIs). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1988) Make the delegation token issued by the MetaStore owned by the right user
[ https://issues.apache.org/jira/browse/HIVE-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1988: -- Fix Version/s: 0.8.0 Status: Patch Available (was: Open) Make the delegation token issued by the MetaStore owned by the right user - Key: HIVE-1988 URL: https://issues.apache.org/jira/browse/HIVE-1988 Project: Hive Issue Type: Bug Components: Metastore, Security, Server Infrastructure Affects Versions: 0.7.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.8.0 Attachments: hive-1988-3.patch, hive-1988.patch The 'owner' of any delegation token issued by the MetaStore is set to the requesting user. When a delegation token is asked by the user himself during a job submission, this is fine. However, in the case where the token is requested for by services (e.g., Oozie), on behalf of the user, the token's owner is set to the user the service is running as. Later on, when the token is used by a MapReduce task, the MetaStore treats the incoming request as coming from Oozie and does operations as Oozie. This means any new directory creations (e.g., create_table) on the hdfs by the MetaStore will end up with Oozie as the owner. Also, the MetaStore doesn't check whether a user asking for a token on behalf of some other user, is actually authorized to act on behalf of that other user. We should start using the ProxyUser authorization in the MetaStore (HADOOP-6510's APIs). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1988) Make the delegation token issued by the MetaStore owned by the right user
[ https://issues.apache.org/jira/browse/HIVE-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1988: -- Attachment: hive-1988-3.patch https://reviews.apache.org/r/528/ is the reviewboard page. Make the delegation token issued by the MetaStore owned by the right user - Key: HIVE-1988 URL: https://issues.apache.org/jira/browse/HIVE-1988 Project: Hive Issue Type: Bug Components: Metastore, Security, Server Infrastructure Affects Versions: 0.7.0 Reporter: Devaraj Das Fix For: 0.8.0 Attachments: hive-1988-3.patch, hive-1988.patch The 'owner' of any delegation token issued by the MetaStore is set to the requesting user. When a delegation token is asked by the user himself during a job submission, this is fine. However, in the case where the token is requested for by services (e.g., Oozie), on behalf of the user, the token's owner is set to the user the service is running as. Later on, when the token is used by a MapReduce task, the MetaStore treats the incoming request as coming from Oozie and does operations as Oozie. This means any new directory creations (e.g., create_table) on the hdfs by the MetaStore will end up with Oozie as the owner. Also, the MetaStore doesn't check whether a user asking for a token on behalf of some other user, is actually authorized to act on behalf of that other user. We should start using the ProxyUser authorization in the MetaStore (HADOOP-6510's APIs). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-1988) Make the delegation token issued by the MetaStore owned by the right user
[ https://issues.apache.org/jira/browse/HIVE-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das reassigned HIVE-1988: - Assignee: Devaraj Das Make the delegation token issued by the MetaStore owned by the right user - Key: HIVE-1988 URL: https://issues.apache.org/jira/browse/HIVE-1988 Project: Hive Issue Type: Bug Components: Metastore, Security, Server Infrastructure Affects Versions: 0.7.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.8.0 Attachments: hive-1988-3.patch, hive-1988.patch The 'owner' of any delegation token issued by the MetaStore is set to the requesting user. When a delegation token is asked by the user himself during a job submission, this is fine. However, in the case where the token is requested for by services (e.g., Oozie), on behalf of the user, the token's owner is set to the user the service is running as. Later on, when the token is used by a MapReduce task, the MetaStore treats the incoming request as coming from Oozie and does operations as Oozie. This means any new directory creations (e.g., create_table) on the hdfs by the MetaStore will end up with Oozie as the owner. Also, the MetaStore doesn't check whether a user asking for a token on behalf of some other user, is actually authorized to act on behalf of that other user. We should start using the ProxyUser authorization in the MetaStore (HADOOP-6510's APIs). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1948) Have audit logging in the Metastore
[ https://issues.apache.org/jira/browse/HIVE-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1948: -- Attachment: audit-log-3.patch Regenerated patch Have audit logging in the Metastore --- Key: HIVE-1948 URL: https://issues.apache.org/jira/browse/HIVE-1948 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.7.0 Attachments: audit-log-2.patch, audit-log-3.patch, audit-log.1.patch, audit-log.patch It would be good to have audit logging in the metastore, similar to Hadoop's NameNode audit logging. This would allow administrators to dig into details about which user performed metadata operations (like create/drop tables/partitions) and from where (IP address). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1948) Have audit logging in the Metastore
[ https://issues.apache.org/jira/browse/HIVE-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12991189#comment-12991189 ] Devaraj Das commented on HIVE-1948: --- https://reviews.apache.org/r/398/ is the reviewboard URL Have audit logging in the Metastore --- Key: HIVE-1948 URL: https://issues.apache.org/jira/browse/HIVE-1948 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.7.0 Attachments: audit-log.1.patch, audit-log.patch It would be good to have audit logging in the metastore, similar to Hadoop's NameNode audit logging. This would allow administrators to dig into details about which user performed metadata operations (like create/drop tables/partitions) and from where (IP address). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1948) Have audit logging in the Metastore
[ https://issues.apache.org/jira/browse/HIVE-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1948: -- Status: Patch Available (was: Open) Submitting patch for review. There is one caveat with this patch - it won't log the IP address of the remote clients when security is enabled in Hive. Making this work means a change in thrift. I have raised THRIFT-1053 for the same. Once THRIFT-1053 is addressed, I will provide a fix (in a different jira) to capture the IP address for the secure case too. Have audit logging in the Metastore --- Key: HIVE-1948 URL: https://issues.apache.org/jira/browse/HIVE-1948 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.7.0 Attachments: audit-log.1.patch, audit-log.patch It would be good to have audit logging in the metastore, similar to Hadoop's NameNode audit logging. This would allow administrators to dig into details about which user performed metadata operations (like create/drop tables/partitions) and from where (IP address). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1948) Have audit logging in the Metastore
[ https://issues.apache.org/jira/browse/HIVE-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1948: -- Attachment: audit-log.1.patch A slightly updated patch. Have audit logging in the Metastore --- Key: HIVE-1948 URL: https://issues.apache.org/jira/browse/HIVE-1948 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.7.0 Attachments: audit-log.1.patch, audit-log.patch It would be good to have audit logging in the metastore, similar to Hadoop's NameNode audit logging. This would allow administrators to dig into details about which user performed metadata operations (like create/drop tables/partitions) and from where (IP address). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HIVE-1943) Metastore operations should do a 'rollback' for HDFS failures
Metastore operations should do a 'rollback' for HDFS failures - Key: HIVE-1943 URL: https://issues.apache.org/jira/browse/HIVE-1943 Project: Hive Issue Type: Bug Reporter: Devaraj Das Fix For: 0.7.0 Currently, metastore operations with associated hdfs operations like drop_partition doesn't do a rollback if the hdfs operation fails. The metastore first updates the metadata, and then tries to do hdfs operation. If the hdfs operation fails for any reason, the data on the hdfs will be orphaned. We should improve the situation. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1927) Fix TestHadoop20SAuthBridge failure on Hudson
[ https://issues.apache.org/jira/browse/HIVE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1927: -- Status: Patch Available (was: Open) Fix TestHadoop20SAuthBridge failure on Hudson - Key: HIVE-1927 URL: https://issues.apache.org/jira/browse/HIVE-1927 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Carl Steinbach Assignee: Devaraj Das Priority: Blocker Fix For: 0.7.0 Attachments: 1927-0.patch The patch for HIVE-1696 is the source a test failure on Hudson. The likely cause of this failure is that the classpath is not being set correctly for the TestHadoop20SAuthBridge test, which depends on the security enhanced version of Hadoop. A couple things to note: * Hive tests are supposed to run against Hadoop 0.20.0 by default. This value is set in the build.properties file in the project top-level. Altering the Hudson job to set hadoop.version to some other value is not a fix for this issue since the Hudson job will then cease to reflect the default behavior of Hive tests. * HIVE-1696 added new secure compile targets to the shims/build.xml file. These targets explicitly set the classpath to include the secure version of Hadoop. The fix for this issue likely involves overriding the test target in shims/build.xml and explicitly setting the classpath in this target to also use the secure version of Hadoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1927) Fix TestHadoop20SAuthBridge failure on Hudson
[ https://issues.apache.org/jira/browse/HIVE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1927: -- Attachment: 1927-0.patch There was an indentation issue in the previous patch. This patch fixes that. Fix TestHadoop20SAuthBridge failure on Hudson - Key: HIVE-1927 URL: https://issues.apache.org/jira/browse/HIVE-1927 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Carl Steinbach Assignee: Devaraj Das Priority: Blocker Fix For: 0.7.0 Attachments: 1927-0.patch, 1927-0.patch The patch for HIVE-1696 is the source a test failure on Hudson. The likely cause of this failure is that the classpath is not being set correctly for the TestHadoop20SAuthBridge test, which depends on the security enhanced version of Hadoop. A couple things to note: * Hive tests are supposed to run against Hadoop 0.20.0 by default. This value is set in the build.properties file in the project top-level. Altering the Hudson job to set hadoop.version to some other value is not a fix for this issue since the Hudson job will then cease to reflect the default behavior of Hive tests. * HIVE-1696 added new secure compile targets to the shims/build.xml file. These targets explicitly set the classpath to include the secure version of Hadoop. The fix for this issue likely involves overriding the test target in shims/build.xml and explicitly setting the classpath in this target to also use the secure version of Hadoop. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12983814#action_12983814 ] Devaraj Das commented on HIVE-1696: --- For the record, I'd like to mention that Pradeep Kamath did a lot of initial work on the patch. Thanks, Pradeep! Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.7.0 Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive-1696-4-with-gen-code.1.patch, hive-1696-4-with-gen-code.patch, hive-1696-4.patch, hive-1696-4.patch, hive_1696.patch, hive_1696.patch, hive_1696_no-thrift.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981814#action_12981814 ] Devaraj Das commented on HIVE-1696: --- Yes, I had forgotten to put it in PA state. Thanks! Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.7.0 Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive-1696-4-with-gen-code.1.patch, hive-1696-4-with-gen-code.patch, hive-1696-4.patch, hive-1696-4.patch, hive_1696.patch, hive_1696.patch, hive_1696_no-thrift.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1862) Revive partition filtering in the Hive MetaStore
[ https://issues.apache.org/jira/browse/HIVE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981887#action_12981887 ] Devaraj Das commented on HIVE-1862: --- Folks, any update on this one? Does it serve our purpose with the limitation that partition values can't have special characters. If so, could it be reviewed. Thanks! Revive partition filtering in the Hive MetaStore Key: HIVE-1862 URL: https://issues.apache.org/jira/browse/HIVE-1862 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Devaraj Das Fix For: 0.7.0 Attachments: HIVE-1862.1.patch.txt, HIVE-1862.2.patch.txt, invoke_runqry.sh, qry, qry-sch.Z, runqry HIVE-1853 downgraded the JDO version. This makes the feature of partition filtering in the metastore unusable. This jira is to keep track of the lost feature and discussing approaches to bring it back. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1696: -- Attachment: hive-1696-4.patch Sorry missed updating the metastore client with the kerberos prefix in the principal name references. This patch fixes that. Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.7.0 Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive-1696-4.patch, hive-1696-4.patch, hive_1696.patch, hive_1696.patch, hive_1696_no-thrift.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1696: -- Attachment: hive-1696-4-with-gen-code.1.patch Thanks Carl! In my earlier patch there was a typo in the testcase, and it was a mistake on my part during the patch generation due to which the typo crept in (due to which the test will fail). I edited Carl's patch and fixed that. Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.7.0 Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive-1696-4-with-gen-code.1.patch, hive-1696-4-with-gen-code.patch, hive-1696-4.patch, hive-1696-4.patch, hive_1696.patch, hive_1696.patch, hive_1696_no-thrift.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1696: -- Attachment: hive-1696-3.patch Attached patch has a testcase that tests the Hive MetaStore client to MetaStore server communication with SASL on delegation tokens. The test is run as part of the test target at the top level. Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Fix For: 0.7.0 Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, hive-1696-3.patch, hive_1696.patch, hive_1696.patch, hive_1696_no-thrift.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-1696: -- Attachment: hive-1696-3-with-gen-code.patch Patch with the generated code.. Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Fix For: 0.7.0 Attachments: hive-1696-1-with-gen-code.patch, hive-1696-1.patch, hive-1696-3-with-gen-code.patch, hive-1696-3.patch, hive_1696.patch, hive_1696.patch, hive_1696_no-thrift.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12972554#action_12972554 ] Devaraj Das commented on HIVE-1696: --- The patch looks fine to me. Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Fix For: 0.7.0 Attachments: hive_1696.patch, hive_1696.patch, hive_1696_no-thrift.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12972171#action_12972171 ] Devaraj Das commented on HIVE-1696: --- Upon a bit more thought, it seems to me that we don't actually need the DelegationTokenManager shim. Since the delegation token stuff is part of Hadoop 20S, we might as well merge these classes/methods in the 20S shim... Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Attachments: hive_1696.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1696) Add delegation token support to metastore
[ https://issues.apache.org/jira/browse/HIVE-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12971936#action_12971936 ] Devaraj Das commented on HIVE-1696: --- I did a walk-thru of the patch. Looks good mostly. One comment I have is that the server should check delegation-token issue/renewal are allowed for kerberos-authenticated users only. This is what is done in the cases of HDFS/MAPREDUCE delegation tokens. Add delegation token support to metastore - Key: HIVE-1696 URL: https://issues.apache.org/jira/browse/HIVE-1696 Project: Hive Issue Type: Sub-task Components: Metastore, Security, Server Infrastructure Reporter: Todd Lipcon Attachments: hive_1696.patch As discussed in HIVE-842, kerberos authentication is only sufficient for authentication of a hive user client to the metastore. There are other cases where thrift calls need to be authenticated when the caller is running in an environment without kerberos credentials. For example, an MR task running as part of a hive job may want to report statistics to the metastore, or a job may be running within the context of Oozie or Hive Server. This JIRA is to implement support of delegation tokens for the metastore. The concept of a delegation token is borrowed from the Hadoop security design - the quick summary is that a kerberos-authenticated client may retrieve a binary token from the server. This token can then be passed to other clients which can use it to achieve authentication as the original user in lieu of a kerberos ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12971514#action_12971514 ] Devaraj Das commented on HIVE-842: -- Thanks Yongqiang for reviewing. Writing a testcase is really difficult - it requires the kerberos setup. We have manually tested this patch at Yahoo!... Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hive Issue Type: New Feature Components: Security, Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Fix For: 0.7.0 Attachments: hive-842.txt, hive-842_2.patch, HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-842) Authentication Infrastructure for Hive
[ https://issues.apache.org/jira/browse/HIVE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12971130#action_12971130 ] Devaraj Das commented on HIVE-842: -- The security related changes look fine to me. Authentication Infrastructure for Hive -- Key: HIVE-842 URL: https://issues.apache.org/jira/browse/HIVE-842 Project: Hive Issue Type: New Feature Components: Security, Server Infrastructure Reporter: Edward Capriolo Assignee: Todd Lipcon Fix For: 0.7.0 Attachments: hive-842.txt, hive-842_2.patch, HiveSecurityThoughts.pdf This issue deals with the authentication (user name,password) infrastructure. Not the authorization components that specify what a user should be able to do. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12934681#action_12934681 ] Devaraj Das commented on HIVE-1526: --- Guys, it will be really appreciated if this patch can be committed now. This blocks the other jiras on Security - HIVE-842 and HIVE-1696. Since Carl and others have done so much work on the thrift patch already, I think it makes sense to have this patch committed now. Thanks! Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.7.0 Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
