[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285976#comment-13285976 ] Francis Liu commented on HBASE-5498: @Andy, I have some code. Sorry for the delay, I got caught up in another project. Let me work on it a bit more and come up with a first draft next week. Does that sound ok? Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_draft.patch Partially finished patch to get feedback. Missing things: * client api for secureBulkLoad, where should this go? Should we have completeBulkLoad API for the client and have that use secure when security is enabled? * doesn't handle failure scenarios. * HFile names have to be numeric so we can create storeFiles. Thinking about renaming each file while permission are being changed. * Tests Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Attachments: HBASE-5498_draft.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291468#comment-13291468 ] Francis Liu commented on HBASE-5498: @Jonathan thanks for the comments will incorporate them in the next patch. * The HCatalog wiki is no longer relevant. The original implementation was being done in HCatalog, then we realized it made more sense to contribute it to HBase. * Yes, probably having the same signature as pre (but with updated family paths) though I haven't thought about how to go about doing that. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-5498_draft.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291471#comment-13291471 ] Francis Liu commented on HBASE-5498: @Ted, the access controller may not need to get a scanner but other coprocessor implementations may. See Andy's first comment. If the user calls LoadIncrementalHFiles directly then the user has to take care of doing chmod, for it to load successfully. As well as doing the procedure described here to make it secure. Maybe we should make LoadIncrementalHFiles use the secure api when security is enabled? Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-5498_draft.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
Francis Liu created HBASE-6224: -- Summary: add Pre and Post coprocessor hooks for BulkLoad Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6224: --- Attachment: HBASE-6224.patch After a discussion with Andy during the HBase BOF. I simplified the signature to just pass the CF-Path pair and let the user create the StoreFile instance if he needs to. I've also added a post hook. add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6225) create secure bulk load endpoint
Francis Liu created HBASE-6225: -- Summary: create secure bulk load endpoint Key: HBASE-6225 URL: https://issues.apache.org/jira/browse/HBASE-6225 Project: HBase Issue Type: Sub-task Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6224: --- Status: Patch Available (was: Open) add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13393646#comment-13393646 ] Francis Liu commented on HBASE-5498: I've split this work into two sub-tasks. Task #1 should be good to go pending any more comments. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-5498_draft.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399616#comment-13399616 ] Francis Liu commented on HBASE-6224: @Andy, looks good to me. Good catch on the bypass and loaded. Should I come up with another patch? add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Components: coprocessors Affects Versions: 0.96.0, 0.94.1 Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224-apurtell.patch, HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399696#comment-13399696 ] Francis Liu commented on HBASE-6224: @Andy, nope nothing to add. BTW can we backport this to 0.94 and 0.92? We're still mulling over wether to pickup 0.92 or 0.94. Though it would be good to get bulkload security covered in the current stable release anyway? add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Components: coprocessors Affects Versions: 0.96.0, 0.94.1 Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224-apurtell.patch, HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6249) Apply ACLs to new bulk load hooks
[ https://issues.apache.org/jira/browse/HBASE-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399764#comment-13399764 ] Francis Liu commented on HBASE-6249: I am already addressing this patch in HBASE-5498. Can I take over this task instead? Apply ACLs to new bulk load hooks - Key: HBASE-6249 URL: https://issues.apache.org/jira/browse/HBASE-6249 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Laxman HBASE-6224 adds coprocessor hooks for bulk loading. This should require table WRITE permission. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399770#comment-13399770 ] Francis Liu commented on HBASE-6224: Is AccessController v2 already in a usable state in 0.94? If not what are the patches that need to go in? add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Components: coprocessors Affects Versions: 0.96.0, 0.94.1 Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224-apurtell-0.94.patch, HBASE-6224-apurtell.patch, HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6249) Apply ACLs to new bulk load hooks
[ https://issues.apache.org/jira/browse/HBASE-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399780#comment-13399780 ] Francis Liu commented on HBASE-6249: @Andy, thanks. My initial patch already did write checks against the list of column families. Will factor it out and put it here prolly this weekend. Apply ACLs to new bulk load hooks - Key: HBASE-6249 URL: https://issues.apache.org/jira/browse/HBASE-6249 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Laxman HBASE-6224 adds coprocessor hooks for bulk loading. This should require table WRITE permission. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6249) Apply ACLs to new bulk load hooks
[ https://issues.apache.org/jira/browse/HBASE-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-6249: -- Assignee: Francis Liu (was: Laxman) Apply ACLs to new bulk load hooks - Key: HBASE-6249 URL: https://issues.apache.org/jira/browse/HBASE-6249 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Francis Liu HBASE-6224 adds coprocessor hooks for bulk loading. This should require table WRITE permission. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13410591#comment-13410591 ] Francis Liu commented on HBASE-5498: Hi Laxman, not yet. Will look into it this week, if that's ok? Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-5498_draft.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412542#comment-13412542 ] Francis Liu commented on HBASE-5498: Hi Laxman, Looks right to me apart from the delegation token. You need to pass an hdfs delegation token because we'd like to impersonate the user when changing permissions on hdfs. Also the path doesn't need to be the full URI. Getting the token should be something like this: FileSystem fs = FileSystem.get(conf); Token? token = fs.getDelegationToken(renewer); Let me know how things go. -Francis Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-5498_draft.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13415834#comment-13415834 ] Francis Liu commented on HBASE-5498: Laxman, what are version are you running it on? I deployed the patch on 0.94 and didn't hit that problem (wasn't sure if trunk was deployable). I hit other problems tho, give me a day or two to iron things out and I'll post a patch. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-5498_draft.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6432) HRegionServer doesn't properly set clusterId in conf
Francis Liu created HBASE-6432: -- Summary: HRegionServer doesn't properly set clusterId in conf Key: HBASE-6432 URL: https://issues.apache.org/jira/browse/HBASE-6432 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 ClusterId is normally set into the passed conf during instantiation of an HTable class. In the case of a HRegionServer this is bypassed and set to default since getMaster() bypasses the class which sets clusterID clusterId since it uses HBaseRPC to create the proxy to create the proxy directly. This becomes a problem with clients (ie within a coprocessor) using delegation tokens for authentication. Since the token's service will be the correct clusterId and while the TokenSelector is looking for one with service default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6432) HRegionServer doesn't properly set clusterId in conf
[ https://issues.apache.org/jira/browse/HBASE-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6432: --- Attachment: HBASE-6432_94.patch a patch for 0.94 to get feedback on the approach. Things changed significant enough in trunk to need a separate patch. I'm hoping to get this backported to 0.94 since it is needed for security. HRegionServer doesn't properly set clusterId in conf Key: HBASE-6432 URL: https://issues.apache.org/jira/browse/HBASE-6432 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-6432_94.patch ClusterId is normally set into the passed conf during instantiation of an HTable class. In the case of a HRegionServer this is bypassed and set to default since getMaster() bypasses the class which sets clusterID clusterId since it uses HBaseRPC to create the proxy to create the proxy directly. This becomes a problem with clients (ie within a coprocessor) using delegation tokens for authentication. Since the token's service will be the correct clusterId and while the TokenSelector is looking for one with service default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_draft_94.patch Laxman, here's a working patch. It incorporates HBASE-6432 which took some time debugging. I still have to address the other comments, some cleanup and TODOs. Let me know if this works for you. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-5498_draft.patch, HBASE-5498_draft_94.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6432) HRegionServer doesn't properly set clusterId in conf
[ https://issues.apache.org/jira/browse/HBASE-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6432: --- Description: ClusterId is normally set into the passed conf during instantiation of an HTable class. In the case of a HRegionServer this is bypassed and set to default since getMaster() since it uses HBaseRPC to create the proxy directly and bypasses the class which retrieves and sets the correct clusterId. This becomes a problem with clients (ie within a coprocessor) using delegation tokens for authentication. Since the token's service will be the correct clusterId and while the TokenSelector is looking for one with service default. was: ClusterId is normally set into the passed conf during instantiation of an HTable class. In the case of a HRegionServer this is bypassed and set to default since getMaster() bypasses the class which sets clusterID clusterId since it uses HBaseRPC to create the proxy to create the proxy directly. This becomes a problem with clients (ie within a coprocessor) using delegation tokens for authentication. Since the token's service will be the correct clusterId and while the TokenSelector is looking for one with service default. HRegionServer doesn't properly set clusterId in conf Key: HBASE-6432 URL: https://issues.apache.org/jira/browse/HBASE-6432 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-6432_94.patch ClusterId is normally set into the passed conf during instantiation of an HTable class. In the case of a HRegionServer this is bypassed and set to default since getMaster() since it uses HBaseRPC to create the proxy directly and bypasses the class which retrieves and sets the correct clusterId. This becomes a problem with clients (ie within a coprocessor) using delegation tokens for authentication. Since the token's service will be the correct clusterId and while the TokenSelector is looking for one with service default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545681#comment-13545681 ] Francis Liu commented on HBASE-5498: I'll finish it, I'll address Ted's comments this week. Sorry for the delay, got caught up with internal releases. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, HBASE-5498_trunk_4.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552514#comment-13552514 ] Francis Liu commented on HBASE-5498: updated trunk patch. also uploaded 0.94 patch reflecting comments from trunk: https://reviews.apache.org/r/8942/ Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, HBASE-5498_trunk_4.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_trunk_5.patch updated patch moved, SecureBulkLoad.proto and generated file to proto module Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, HBASE-5498_trunk_4.patch, HBASE-5498_trunk_5.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_94_3.patch Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, HBASE-5498_94_3.patch, HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, HBASE-5498_trunk_4.patch, HBASE-5498_trunk_5.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554191#comment-13554191 ] Francis Liu commented on HBASE-5498: Thanks Andy and Ted! Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, HBASE-5498_94_3.patch, HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, HBASE-5498_trunk_4.patch, HBASE-5498_trunk_5.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Release Note: This feature adds a new optional configuration parameter: hbase.bulkload.staging.dir This defines the path on DFS that HBase will use to create random/secret directories under. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, HBASE-5498_94_3.patch, HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, HBASE-5498_trunk_4.patch, HBASE-5498_trunk_5.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7043) Region Server Group CLI commands
[ https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7043: --- Attachment: HBASE-7043_94_3.patch Updated patch. I had to backport some code from trunk to support modifying table properties. Sorry for the delay. Region Server Group CLI commands Key: HBASE-7043 URL: https://issues.apache.org/jira/browse/HBASE-7043 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-6721_94_2.patch, HBASE-7043_94_2.patch, HBASE-7043_94_3.patch, HBASE-7043_94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571198#comment-13571198 ] Francis Liu commented on HBASE-6721: Will update documentation tomorrow. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6721: --- Attachment: HBASE-6721_94_6.patch updated patch, also updated review board. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7770) minor integration test framework fixes
Francis Liu created HBASE-7770: -- Summary: minor integration test framework fixes Key: HBASE-7770 URL: https://issues.apache.org/jira/browse/HBASE-7770 Project: HBase Issue Type: Bug Reporter: Francis Liu Priority: Minor - made FileSystem on HBaseTestingUtil.createMulti() not expect mini cluster - added check if server is not running before deciding to restore a server -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7770) minor integration test framework fixes
[ https://issues.apache.org/jira/browse/HBASE-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7770: --- Attachment: HBASE-7770_94.patch minor integration test framework fixes -- Key: HBASE-7770 URL: https://issues.apache.org/jira/browse/HBASE-7770 Project: HBase Issue Type: Bug Reporter: Francis Liu Priority: Minor Attachments: HBASE-7770_94.patch - made FileSystem on HBaseTestingUtil.createMulti() not expect mini cluster - added check if server is not running before deciding to restore a server -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7770) minor integration test framework fixes
[ https://issues.apache.org/jira/browse/HBASE-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7770: --- Attachment: HBASE-7770.patch minor integration test framework fixes -- Key: HBASE-7770 URL: https://issues.apache.org/jira/browse/HBASE-7770 Project: HBase Issue Type: Bug Reporter: Francis Liu Priority: Minor Attachments: HBASE-7770_94.patch, HBASE-7770.patch - made FileSystem on HBaseTestingUtil.createMulti() not expect mini cluster - added check if server is not running before deciding to restore a server -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7770) minor integration test framework fixes
[ https://issues.apache.org/jira/browse/HBASE-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7770: --- Assignee: Francis Liu Status: Patch Available (was: Open) minor integration test framework fixes -- Key: HBASE-7770 URL: https://issues.apache.org/jira/browse/HBASE-7770 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Priority: Minor Attachments: HBASE-7770_94.patch, HBASE-7770.patch - made FileSystem on HBaseTestingUtil.createMulti() not expect mini cluster - added check if server is not running before deciding to restore a server -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7771) Secure HBase Client in MR job causes tasks to wait forever
Francis Liu created HBASE-7771: -- Summary: Secure HBase Client in MR job causes tasks to wait forever Key: HBASE-7771 URL: https://issues.apache.org/jira/browse/HBASE-7771 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.5 This seems to be a regression caused by HBASE-4791 wherein we check if secure zookeeper is enabled and if so we make use of saslLatch to verify security handshake is completed. But in the case of MR, we won't be negotiating a secure connection thus we end up waiting forever for the saslLatch. Since the bug the saslLatch workaround is trying to fix (ZOOKEEPER-1437) is already fixed in zookeeper-3.4.5. Removal of the workaround fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7771) Secure HBase Client in MR job causes tasks to wait forever
[ https://issues.apache.org/jira/browse/HBASE-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7771: --- Attachment: HBASE-7771_94.patch Secure HBase Client in MR job causes tasks to wait forever -- Key: HBASE-7771 URL: https://issues.apache.org/jira/browse/HBASE-7771 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.5 Attachments: HBASE-7771_94.patch This seems to be a regression caused by HBASE-4791 wherein we check if secure zookeeper is enabled and if so we make use of saslLatch to verify security handshake is completed. But in the case of MR, we won't be negotiating a secure connection thus we end up waiting forever for the saslLatch. Since the bug the saslLatch workaround is trying to fix (ZOOKEEPER-1437) is already fixed in zookeeper-3.4.5. Removal of the workaround fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7771) Secure HBase Client in MR job causes tasks to wait forever
[ https://issues.apache.org/jira/browse/HBASE-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7771: --- Status: Patch Available (was: Open) Secure HBase Client in MR job causes tasks to wait forever -- Key: HBASE-7771 URL: https://issues.apache.org/jira/browse/HBASE-7771 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.5 Attachments: HBASE-7771_94.patch, HBASE-7771.patch This seems to be a regression caused by HBASE-4791 wherein we check if secure zookeeper is enabled and if so we make use of saslLatch to verify security handshake is completed. But in the case of MR, we won't be negotiating a secure connection thus we end up waiting forever for the saslLatch. Since the bug the saslLatch workaround is trying to fix (ZOOKEEPER-1437) is already fixed in zookeeper-3.4.5. Removal of the workaround fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7771) Secure HBase Client in MR job causes tasks to wait forever
[ https://issues.apache.org/jira/browse/HBASE-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7771: --- Attachment: HBASE-7771.patch Secure HBase Client in MR job causes tasks to wait forever -- Key: HBASE-7771 URL: https://issues.apache.org/jira/browse/HBASE-7771 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.5 Attachments: HBASE-7771_94.patch, HBASE-7771.patch This seems to be a regression caused by HBASE-4791 wherein we check if secure zookeeper is enabled and if so we make use of saslLatch to verify security handshake is completed. But in the case of MR, we won't be negotiating a secure connection thus we end up waiting forever for the saslLatch. Since the bug the saslLatch workaround is trying to fix (ZOOKEEPER-1437) is already fixed in zookeeper-3.4.5. Removal of the workaround fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
Francis Liu created HBASE-7772: -- Summary: clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7772: --- Attachment: HBASE-7772_94.patch This is more a short term fix as we set the cluster id whenever a token is added to the job. It seems there needs to be some major refactoring to come up with a more holistic solution. clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-7772_94.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571588#comment-13571588 ] Francis Liu commented on HBASE-7772: Yes, I believe this may be what caused the regression. No remote clusters are involved in my case. Simple way to try this out is run PerformanceEvaluation tests in secure mode. clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-7772_94.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7772: --- Status: Patch Available (was: Open) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-7772_94.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7770) minor integration test framework fixes
[ https://issues.apache.org/jira/browse/HBASE-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571603#comment-13571603 ] Francis Liu commented on HBASE-7770: I wanted to keep things simple and give the api more flexibility. Either way works for me, should I change it? minor integration test framework fixes -- Key: HBASE-7770 URL: https://issues.apache.org/jira/browse/HBASE-7770 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Priority: Minor Attachments: HBASE-7770_94.patch, HBASE-7770.patch - made FileSystem on HBaseTestingUtil.createMulti() not expect mini cluster - added check if server is not running before deciding to restore a server -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7772: --- Fix Version/s: 0.94.5 0.96.0 clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7772_94.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7772: --- Attachment: HBASE-7772.patch port of trunk patch. I have not tested this. clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7772_94.patch, HBASE-7772.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571633#comment-13571633
]
Francis Liu commented on HBASE-6721:
{quote}My main question is - what is the state of the trunk patch?{quote}
Our main focus was to get something released internally thus it's a bit behind.
Now that we have things more or less stabilized we'll have time to get this
done.
{quote}Another one may be (may be - due to not having seen the rest of the
patch yet) - why is some JSON serialization used?{quote}
It was an easy way of not having to deal with actually serializing the data.
I'm open to suggestions. For trunk we can use PB.
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721_94_2.patch, HBASE-6721_94_3.patch,
HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch,
HBASE-6721_94_6.patch, HBASE-6721_94.patch, HBASE-6721_94.patch,
HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7771) Secure HBase Client in MR job causes tasks to wait forever
[ https://issues.apache.org/jira/browse/HBASE-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571716#comment-13571716 ] Francis Liu commented on HBASE-7771: [~te...@apache.org] The bug is in the client-side so they will be using 3.4.5 not unless they're doing their own packaging. Secure HBase Client in MR job causes tasks to wait forever -- Key: HBASE-7771 URL: https://issues.apache.org/jira/browse/HBASE-7771 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: 7771-0.94-combined.txt, 7771-combined.txt, HBASE-7771_94.patch, HBASE-7771-addendum-v0.patch, HBASE-7771.patch This seems to be a regression caused by HBASE-4791 wherein we check if secure zookeeper is enabled and if so we make use of saslLatch to verify security handshake is completed. But in the case of MR, we won't be negotiating a secure connection thus we end up waiting forever for the saslLatch. Since the bug the saslLatch workaround is trying to fix (ZOOKEEPER-1437) is already fixed in zookeeper-3.4.5. Removal of the workaround fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571831#comment-13571831 ] Francis Liu commented on HBASE-7772: I can easily reproduce this problem by running the PerformanceEvaluation tool to launch MR jobs. I'm running on hadoop-0.23 BTW. Let me try and dig deeper and figure out the cause. clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7772_94.patch, HBASE-7772.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571833#comment-13571833 ] Francis Liu commented on HBASE-7772: Here's a stacktrace, perhaps the issue is isolated with HBaseAdmin: 2013-02-05 22:45:30,361 FATAL [IPC Server handler 25 on 60437] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1348093718159_1526_m_46_3 - exited : java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) at org.apache.hadoop.hbase.security.User.call(User.java:590) at org.apache.hadoop.hbase.security.User.access$700(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:444) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:203) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:291) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974) at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:104) at $Proxy11.getProtocolVersion(Unknown Source) at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:146) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:708) at org.apache.hadoop.hbase.client.HBaseAdmin.init(HBaseAdmin.java:113) at org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:788) at org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testSetup(PerformanceEvaluation.java:998) at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:808) at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1141) at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:448) at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:401) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:156) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:177) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:85) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:284) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:281) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at
[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571887#comment-13571887 ] Francis Liu commented on HBASE-7772: Took a quick look this is what seems to be happening: - SecureRPCengine initialization is created as part of HConnectionImplmentation's constructor - SecureClient (subclass of HBaseClient), is created as part of SecureRPCEngine initialization - HBaseClient.clusterId is assigned from conf (clusterId = conf.get(CLUSTER_ID,default) as part of HBaseClient's constructor - SaslClient is the consumer of HBaseClient.clusterId to determine if a delegation token can be used - HConnectionImplementation.ensureZookeeperTrackers() is responsible for setting the clusterId in conf The problem here is the sequence in which things happen. Since SecureRPCEngine is created before HConnectionImplementation.ensureZookeeperTrackers() is called. HBaseClient.clusterId downstream will be set to default. clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7772_94.patch, HBASE-7772.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571901#comment-13571901 ] Francis Liu commented on HBASE-7772: A better fix could be to retrieve and set clusterId in HConnectionImplementation's constructor. Though I'm not entirely familiar with the client code. Thoughts? clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7772_94.patch, HBASE-7772.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7771) Secure HBase Client in MR job causes tasks to wait forever
[ https://issues.apache.org/jira/browse/HBASE-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571917#comment-13571917 ] Francis Liu commented on HBASE-7771: [~ekoontz] Thanks for that contribution [~lhofhansl] Just reiterating what Eugene said. No upgrade of servers are necessary. The only change is users running with secure zookeeper will have to upgrade their _clients_ to 3.4.5 with this patch. Secure HBase Client in MR job causes tasks to wait forever -- Key: HBASE-7771 URL: https://issues.apache.org/jira/browse/HBASE-7771 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: 7771-0.94-combined.txt, 7771-combined.txt, HBASE-7771_94.patch, HBASE-7771-addendum-v0.patch, HBASE-7771.patch This seems to be a regression caused by HBASE-4791 wherein we check if secure zookeeper is enabled and if so we make use of saslLatch to verify security handshake is completed. But in the case of MR, we won't be negotiating a secure connection thus we end up waiting forever for the saslLatch. Since the bug the saslLatch workaround is trying to fix (ZOOKEEPER-1437) is already fixed in zookeeper-3.4.5. Removal of the workaround fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7771) Secure HBase Client in MR job causes tasks to wait forever
[ https://issues.apache.org/jira/browse/HBASE-7771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571932#comment-13571932 ] Francis Liu commented on HBASE-7771: Though the 3.4.5 upgrade is already part of our tarball distribution. Secure HBase Client in MR job causes tasks to wait forever -- Key: HBASE-7771 URL: https://issues.apache.org/jira/browse/HBASE-7771 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: 7771-0.94-combined.txt, 7771-combined.txt, HBASE-7771_94.patch, HBASE-7771-addendum-v0.patch, HBASE-7771.patch This seems to be a regression caused by HBASE-4791 wherein we check if secure zookeeper is enabled and if so we make use of saslLatch to verify security handshake is completed. But in the case of MR, we won't be negotiating a secure connection thus we end up waiting forever for the saslLatch. Since the bug the saslLatch workaround is trying to fix (ZOOKEEPER-1437) is already fixed in zookeeper-3.4.5. Removal of the workaround fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572033#comment-13572033 ] Francis Liu commented on HBASE-7772: [~ghelmling] I believe the reason RowCounter works is because it uses TableInputFormat and when TIF.setConf is called. It creates an instance of HTable which should set clsuterId prior to job submission. Great, I can verify the patch once it's up. clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7772_94.patch, HBASE-7772.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called
[ https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572068#comment-13572068 ] Francis Liu commented on HBASE-7772: +1 works for me clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called Key: HBASE-7772 URL: https://issues.apache.org/jira/browse/HBASE-7772 Project: HBase Issue Type: Bug Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7772_94.patch, HBASE-7772.patch, HBASE-7772-v2_0.94.patch, HBASE-7772-v2.patch clusterId is not set in the job's conf correctly if only TableMapReduceUtil.initCredentials() is called and thus fails to include the token when using an hbase client in any of the job's tasks. This is a regression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7042) Master Coprocessor Endpoint
Francis Liu created HBASE-7042: -- Summary: Master Coprocessor Endpoint Key: HBASE-7042 URL: https://issues.apache.org/jira/browse/HBASE-7042 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-7042_94.patch Having support for a master coprocessor endpoint would enable developers to easily extended HMaster functionality/features. As is the case for region server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7042) Master Coprocessor Endpoint
[ https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7042: --- Attachment: HBASE-7042_94.patch Master Coprocessor Endpoint --- Key: HBASE-7042 URL: https://issues.apache.org/jira/browse/HBASE-7042 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7042_94.patch Having support for a master coprocessor endpoint would enable developers to easily extended HMaster functionality/features. As is the case for region server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[ https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483045#comment-13483045 ] Francis Liu commented on HBASE-7042: Putting up a 0.94 patch for initial comments. Master Coprocessor Endpoint --- Key: HBASE-7042 URL: https://issues.apache.org/jira/browse/HBASE-7042 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7042_94.patch Having support for a master coprocessor endpoint would enable developers to easily extended HMaster functionality/features. As is the case for region server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6721: --- Attachment: HBASE-6721_94.patch 0.94 patch for initial review RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721_94.patch, HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7043) Region Server Group CLI commands
Francis Liu created HBASE-7043: -- Summary: Region Server Group CLI commands Key: HBASE-7043 URL: https://issues.apache.org/jira/browse/HBASE-7043 Project: HBase Issue Type: Sub-task Reporter: Francis Liu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7043) Region Server Group CLI commands
[ https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7043: --- Attachment: HBASE-7043_94.patch Region Server Group CLI commands Key: HBASE-7043 URL: https://issues.apache.org/jira/browse/HBASE-7043 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7043_94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7043) Region Server Group CLI commands
[ https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-7043: -- Assignee: Francis Liu Region Server Group CLI commands Key: HBASE-7043 URL: https://issues.apache.org/jira/browse/HBASE-7043 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7043_94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7043) Region Server Group CLI commands
[ https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483056#comment-13483056 ] Francis Liu commented on HBASE-7043: 0.94 initial patch for review Region Server Group CLI commands Key: HBASE-7043 URL: https://issues.apache.org/jira/browse/HBASE-7043 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7043_94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6721: --- Status: Patch Available (was: Open) 0.94 patch for initial review, We decided to combine the patch of two subtasks into one. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721_94.patch, HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6723) Implement RegionServer Group Based Balancer
[ https://issues.apache.org/jira/browse/HBASE-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu resolved HBASE-6723. Resolution: Invalid Implement RegionServer Group Based Balancer --- Key: HBASE-6723 URL: https://issues.apache.org/jira/browse/HBASE-6723 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Re-purposing this Jira after the discussion last week. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6837) RegionServer Groups corpcoessor apis
[ https://issues.apache.org/jira/browse/HBASE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu resolved HBASE-6837. Resolution: Invalid RegionServer Groups corpcoessor apis Key: HBASE-6837 URL: https://issues.apache.org/jira/browse/HBASE-6837 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7042) Master Coprocessor Endpoint
[ https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7042: --- Status: Patch Available (was: Open) Master Coprocessor Endpoint --- Key: HBASE-7042 URL: https://issues.apache.org/jira/browse/HBASE-7042 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7042_94.patch Having support for a master coprocessor endpoint would enable developers to easily extended HMaster functionality/features. As is the case for region server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7043) Region Server Group CLI commands
[ https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7043: --- Status: Patch Available (was: Open) Region Server Group CLI commands Key: HBASE-7043 URL: https://issues.apache.org/jira/browse/HBASE-7043 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7043_94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6721: --- Attachment: HBASE-6721_94.patch RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721_94.patch, HBASE-6721_94.patch, HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483464#comment-13483464
]
Francis Liu commented on HBASE-6721:
[~yuzhih...@gmail.com]
{quote}
What does the value in serversInTransition map represent?
{quote}
It represents servers that are being moved from one group to another.
{quote}
Can you tell me what the body is supposed to achieve ?
Back to GroupAdminEndpoint:
{quote}
Retrieveing the balancer during start() returns null. Thus I have to retrieve
it lazily as needed.
{quote}
Does GroupInfoManager belong to balancer ? The above is probably the longest
indirection I have ever seen
{quote}
We had to do this since we didn't want to touch AssignmentManager as much as
possible :)
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721_94.patch, HBASE-6721_94.patch,
HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483471#comment-13483471
]
Francis Liu commented on HBASE-6721:
{quote}
We had to do this since we didn't want to touch AssignmentManager as much as
possible
{quote}
As an alternative, we can add a getBalancer() Method to MasterServices.
Thoughts?
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721_94.patch, HBASE-6721_94.patch,
HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483642#comment-13483642
]
Francis Liu commented on HBASE-6721:
{quote}
I was asking about the following line of code:
found = found || servers.contains(server);
It seems to be condition checking.
{quote}
Yeah it's basically checking if the list of servers to be moved is already in
the ServersInTransition list, meaning it is already being moved so we shouldn't
allow that.
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721_94.patch, HBASE-6721_94.patch,
HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483768#comment-13483768
]
Francis Liu commented on HBASE-6721:
Thanks for the comments [~eclark]
{quote}
I couldn't seem to get it to compile for me on 0.94
{quote}
Did you apply the patches attached in the subtasks prior to apply this patch?
If you'd like them all in one single patch I can do that as well.
{quote}
There seem to be a bunch of formatting changes that aren't needed.
{quote}
Will clean that up in the next update.
{quote}
Passing in the preferred server into the load balancer on randomAssignment is
messy. If we know the preferred server why call this function at all ?
{quote}
Good point, will remove that argument.
{quote}
The balancer is a public interface and we can't make changes to it in a minor
release. And this patch won't apply to trunk.
{quote}
I see, we can make it binary compatible at least by supporting both interfaces
if you're amenable to that. We're planning on getting 0.94 into production and
it'd be great if we didn't have a lot of custom patches on top of it.
{quote}
With this many interfaces and classes it might make sense to move them into a
namespace.
{quote}
Will look into doing this, my main concern is if there any dependencies to
package private methods.
{quote}
Why a co-processor and not build it in ?
Security was done that was because it can be added or removed. As the patch
is that's not really possible
This makes a lot of changes in core code for something that is a
co-processor.
{quote}
As part of the design, HBase should run fine without the group based classes
enabled (endpoint, balancer, etc). If it is not that case then that's a bug. As
for some code changes in core code. Some may be unavoidable, but we could
probably still make it less invasive (ie remove the EventHandler changes).
Having said that, I don't mind if the community would like to have this fully
integrated into HBase, just let us know.
{quote}
Don't create a DefaultLoadBalancer in GroupBasedLoadBalancer. The balancer was
made pluggable and that feature shouldn't go away.
{quote}
The balancer is still pluggable it's just not pluggable for the
GroupBasedLoadBalancer. Though should be ok to make that pluggable as well.
{quote}
HTableDescriptor seems like the correct location for info about the table if
you don't want to put that data into meta.
{quote}
Yes, we have group affiliation store as a table property. Though group
information is stored on hdfs.
{quote}
putting things into the filesystem seems like the wrong way to do it. There are
just so many different moving parts with getting things from hdfs with caching
and cache invalidation, and edge cases on failure.
{quote}
I see, were do you suggest we put it? Zookeeper? We mainly had it in HDFS since
ZK, seemed to be the place to store only ephemeral data? Putting the data in
tables would be a lot more complex and would require more core code change.
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721_94.patch, HBASE-6721_94.patch,
HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483785#comment-13483785
]
Francis Liu commented on HBASE-6721:
{quote}
You're missing the change where null plans are no longer queued which comes
about because of this patch.
{quote}
We needed this change to prevent regions from being assigned to region servers
they don't belong to. We can continue to recognize null, we just need another
way to prevent regions from being assigned to the wrong group of region
servers. One option is to have a dead/bogus server as part of the plan if no
online servers are available for a given group, this way it eventually gets
reassigned once a live server is up. Would that work?
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721_94.patch, HBASE-6721_94.patch,
HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483849#comment-13483849
]
Francis Liu commented on HBASE-6721:
{quote}
I would prefer something like security has. We have a perfectly usable
key/value storage system no need to use a different storage path.
{quote}
That was what we had originally in mind, but decided to punt on it due to the
complexity. The main problem here is during startup before the catalog tables
get assigned we would need to know what group the tables are members of. And
since the group table isn't assigned yet we have to get creative. In my mind
one way to do that is manually read the region on hdfs to retrieve the group
information. Does that sound feasible?
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721_94.patch, HBASE-6721_94.patch,
HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7043) Region Server Group CLI commands
[
https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484342#comment-13484342
]
Francis Liu commented on HBASE-7043:
{quote}
1) Comment is invalid.
2) Similar functionality is in trunk already; see CONFIG. Or is it intended for
something else?
{quote}
Good catch, can we backport this functionality?
Region Server Group CLI commands
Key: HBASE-7043
URL: https://issues.apache.org/jira/browse/HBASE-7043
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu
Fix For: 0.96.0
Attachments: HBASE-7043_94.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7043) Region Server Group CLI commands
[
https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484351#comment-13484351
]
Francis Liu commented on HBASE-7043:
{quote}
Also the patch is for 0.94 but fixed version says 0.96. Is is this for 0.94?
{quote}
Yep, we will be creating an 0.96 patch. 0.94 is the version we currently plan
on using this feature with thus hoping it can be backported.
Region Server Group CLI commands
Key: HBASE-7043
URL: https://issues.apache.org/jira/browse/HBASE-7043
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu
Fix For: 0.96.0
Attachments: HBASE-7043_94.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
Francis Liu created HBASE-7066: -- Summary: Some HMaster coprocessor exceptions are being swallowed in try catch blocks Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors Reporter: Francis Liu Fix For: 0.96.0, 0.94.4 This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7066: --- Attachment: HBASE-7066_trunk.patch Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors Reporter: Francis Liu Fix For: 0.96.0, 0.94.4 Attachments: HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7066: --- Attachment: HBASE-7066_94.patch 0.94 backport, I didn't fix the hooks in the balancer() method as this would change the method signature and break backward compatiblity. Let me know if I should. Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors Reporter: Francis Liu Fix For: 0.96.0, 0.94.4 Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-7066: -- Assignee: Francis Liu Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.4 Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7066: --- Status: Patch Available (was: Open) Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.4 Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487526#comment-13487526 ] Francis Liu commented on HBASE-7066: Ted, Fixed the nit. Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.94.2, 0.96.0 Reporter: Francis Liu Assignee: Francis Liu Priority: Critical Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-7066: --- Attachment: HBASE-7066_trunk.patch Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.94.2, 0.96.0 Reporter: Francis Liu Assignee: Francis Liu Priority: Critical Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[ https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487560#comment-13487560 ] Francis Liu commented on HBASE-7042: Ted, not sure why this would break rolling restart? Master Coprocessor Endpoint --- Key: HBASE-7042 URL: https://issues.apache.org/jira/browse/HBASE-7042 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7042_94.patch Having support for a master coprocessor endpoint would enable developers to easily extended HMaster functionality/features. As is the case for region server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487571#comment-13487571 ] Francis Liu commented on HBASE-6721: In the current implementation we handle META the same way as all the other tables. Since we just load the group information from hdfs. If the group information was stored on a table we won't be able to access the group information through normal means until after we assign ROOT, META and the group table itself. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721_94.patch, HBASE-6721_94.patch, HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7043) Region Server Group CLI commands
[ https://issues.apache.org/jira/browse/HBASE-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487572#comment-13487572 ] Francis Liu commented on HBASE-7043: What keeps it from getting into 0.94? Is it the new method in HMaster? Since we're only adding a method that should keep things binary compatible to older versions at least. Region Server Group CLI commands Key: HBASE-7043 URL: https://issues.apache.org/jira/browse/HBASE-7043 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7043_94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488198#comment-13488198 ] Francis Liu commented on HBASE-7066: Stack, IMHO it really depends on what the coprocessor wants. In the context of security, it'd be safer to err on the side of caution if the security module is unable to authorize a request, as this behavior may be exploited by a malicious user. In general though we probably need a means for the coprocessor to communicate what can and can't be ignored. Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.94.2, 0.96.0 Reporter: Francis Liu Assignee: Francis Liu Priority: Critical Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488215#comment-13488215 ] Francis Liu commented on HBASE-7066: Ted, I did not change the signatures for HMaster in 0.94 as Ram suggested. So that patch should be ok. Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.94.2, 0.96.0 Reporter: Francis Liu Assignee: Francis Liu Priority: Critical Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[ https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488384#comment-13488384 ] Francis Liu commented on HBASE-7042: Gary, Yep, I'll be working on a trunk patch as well. I did 0.94 first as it's what we'll be deploying to production. This change shouldn't break binary backward compatibity for 0.94? Master Coprocessor Endpoint --- Key: HBASE-7042 URL: https://issues.apache.org/jira/browse/HBASE-7042 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7042_94.patch Having support for a master coprocessor endpoint would enable developers to easily extended HMaster functionality/features. As is the case for region server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[
https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488394#comment-13488394
]
Francis Liu commented on HBASE-7042:
{quote}
Can we use the same Exec and ExecResponse classes for both master and RS
endpoint invocations? (I.e. get rid of MasterExec and MasterExecResult?)
{quote}
It's not possible since the region classes contains relations to
regions/row/etc.
{quote}
The patch handles system coprocessors, those registered by Configuration, but
what about table coprocessors? Table coprocessors are loaded as dictated by
table attributes. Should the master load those when creating or updating table
schema and discovers that coprocessor load directives now exist in the schema?
We wouldn't need to load a table coprocessor into the master if the table is
disabled, but once the table is enabled, then we would?
{quote}
Do we really need a table specific MasterEndpoint? For Region's it was really
useful since it'd enable you to do distributed computation across regions apart
from being able to extend Region/RegionServer functionality. In the master's
case, most use cases would be to extend the master's functionality?
Master Coprocessor Endpoint
---
Key: HBASE-7042
URL: https://issues.apache.org/jira/browse/HBASE-7042
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu
Fix For: 0.96.0
Attachments: HBASE-7042_94.patch
Having support for a master coprocessor endpoint would enable developers to
easily extended HMaster functionality/features. As is the case for region
server grouping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488818#comment-13488818 ] Francis Liu commented on HBASE-7066: More specifically if AccessController.preShutdown() throws an IOException, shutdown should fail since we cannot truly determine whether the shutdown request is authorized and thus should err on the side of caution and fail the request. There may be instances that preShutdown() hook exceptions can be ignored but that is not the case for AccessController which comes packaged with HBase thus it would make sense to guarantee that it works as advertised. We can address the previous scenario by fixing the coprocessor framework. Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.94.2, 0.96.0 Reporter: Francis Liu Assignee: Francis Liu Priority: Critical Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488822#comment-13488822 ] Francis Liu commented on HBASE-7066: Lars, if you agree that this fix is the right way to go. Can we get this into 0.94.3? Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.94.2, 0.96.0 Reporter: Francis Liu Assignee: Francis Liu Priority: Critical Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[
https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488838#comment-13488838
]
Francis Liu commented on HBASE-7042:
{quote}
The issue I see with supporting only system coprocessors is it requires
deployment by way of XML config update as the sole option, which requires a
process restart. Table CPs can be deployed to HDFS and then loaded dynamically
with a schema update without needing to stop any process. Not supporting this
latter option on the master seems a shortcut and limiting.
{quote}
Don't we already have a precedent for this with MasterObserver? From a use case
perspective it would seem to me that Master Coprocessors are cluster owner
decisions and are prolly fewer and slower moving than region coprocessors which
are owned by table owners. In the former case updating and restarting the
master should be acceptable?
Master Coprocessor Endpoint
---
Key: HBASE-7042
URL: https://issues.apache.org/jira/browse/HBASE-7042
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu
Fix For: 0.96.0
Attachments: HBASE-7042_94.patch
Having support for a master coprocessor endpoint would enable developers to
easily extended HMaster functionality/features. As is the case for region
server grouping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[ https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488845#comment-13488845 ] Francis Liu commented on HBASE-7042: Though I believe your concern can be addressed by making system coprocessors reloadable. Which I think we should do for both master and region coprocessors anyway. This we can address in a separate jira? Master Coprocessor Endpoint --- Key: HBASE-7042 URL: https://issues.apache.org/jira/browse/HBASE-7042 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-7042_94.patch Having support for a master coprocessor endpoint would enable developers to easily extended HMaster functionality/features. As is the case for region server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[
https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488849#comment-13488849
]
Francis Liu commented on HBASE-7042:
{quote}
It is possible to reuse Exec and ExecResult. Just ignore / don't set the row
fields etc. Having separate classes for master exec only is not a clean API in
my opinion.
{quote}
Apart from being a bit clunky I made them separate so that each class can
evolve without master/region usages stepping/confusing on each other. If you
think this is ok I'm fine with reusing the Exec and ExecResult.
Master Coprocessor Endpoint
---
Key: HBASE-7042
URL: https://issues.apache.org/jira/browse/HBASE-7042
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Francis Liu
Fix For: 0.96.0
Attachments: HBASE-7042_94.patch
Having support for a master coprocessor endpoint would enable developers to
easily extended HMaster functionality/features. As is the case for region
server grouping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_trunk_4.patch protobuff version for trunk Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.4 Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, HBASE-5498_trunk_4.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13489309#comment-13489309 ] Francis Liu commented on HBASE-5498: Though I'm wondering why it's necessary to add throws Throwable to HTable.coprocessorService? Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0, 0.94.4 Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, HBASE-5498_trunk_4.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13489646#comment-13489646 ] Francis Liu commented on HBASE-7066: Stack, I see what you're trying to get at, we can have AccessController wrap the exceptions it would have thrown in an AccessDeniedException so it won't get swallowed in shutdown(). The only potential concern here then is the CoprocessorHost messing up though in which case it prolly won't be throwing an IOE so that should acceptable. Ted, for 0.94 we would have to do some reflection since the Exception is the security module, or actually move the exception. We can't change the signature since that would break backward compatilibity. Have we ever considered making security first class in 0.96 (not a coprocessor, though still switchable)? More and more it seems to me that security cannot be treated as a bolt-on extension as we have seen handling failures cannot be treated the same way as other extensions. We are already considering changing a public method signature to throw a security exception. Some HMaster coprocessor exceptions are being swallowed in try catch blocks --- Key: HBASE-7066 URL: https://issues.apache.org/jira/browse/HBASE-7066 Project: HBase Issue Type: Bug Components: Coprocessors, security Affects Versions: 0.94.2, 0.96.0 Reporter: Francis Liu Assignee: Francis Liu Priority: Critical Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, HBASE-7066_trunk.patch This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
