[jira] [Commented] (HBASE-5827) [Coprocessors] Observer notifications on exceptions
[
https://issues.apache.org/jira/browse/HBASE-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257483#comment-13257483
]
Andrew Purtell commented on HBASE-5827:
---
I can try wrapping the code between pre and post hooks with try/catch in a way
that doesn't change indenting inbetween.
[Coprocessors] Observer notifications on exceptions
---
Key: HBASE-5827
URL: https://issues.apache.org/jira/browse/HBASE-5827
Project: HBase
Issue Type: Improvement
Components: coprocessors
Reporter: Andrew Purtell
Benjamin Busjaeger wrote on dev@:
{quote}
Is there a reason that RegionObservers are not notified when a get/put/delete
fails? Suppose I maintain some (transient) state in my Coprocessor that is
created during preGet and discarded during postGet. If the get fails, postGet
is not invoked, so I cannot remove the state.
If there is a good reason, is there any other way to achieve the same thing?
If not, would it be possible to add something the snippet below to the code
base?
{code}
// pre-get CP hook
if (withCoprocessor (coprocessorHost != null)) {
if (coprocessorHost.preGet(get, results)) {
return results;
}
}
+try{
...
+} catch (Throwable t) {
+// failed-get CP hook
+if (withCoprocessor (coprocessorHost != null)) {
+ coprocessorHost.failedGet(get, results);
+}
+rethrow t;
+}
// post-get CP hook
if (withCoprocessor (coprocessorHost != null)) {
coprocessorHost.postGet(get, results);
}
{code}
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5787) Table owner can't disable/delete its own table
[
https://issues.apache.org/jira/browse/HBASE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257104#comment-13257104
]
Andrew Purtell commented on HBASE-5787:
---
This seems fine. The initial implementation had the notion of requiring ADMIN
for any table op that has global cluster implications, but then went back and
changed that to allow at least users to create tables if they had CREATE
permission. The delete case seems a bug. All around an area that required
improvement.
Table owner can't disable/delete its own table
--
Key: HBASE-5787
URL: https://issues.apache.org/jira/browse/HBASE-5787
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Labels: acl, security
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-5787-tests-wrong-names.patch, HBASE-5787-v0.patch,
HBASE-5787-v1.patch
An user with CREATE privileges can create a table, but can not disable it,
because disable operation require ADMIN privileges. Also if a table is
already disabled, anyone can remove it.
{code}
public void preDeleteTable(ObserverContextMasterCoprocessorEnvironment c,
byte[] tableName) throws IOException {
requirePermission(Permission.Action.CREATE);
}
public void preDisableTable(ObserverContextMasterCoprocessorEnvironment c,
byte[] tableName) throws IOException {
/* TODO: Allow for users with global CREATE permission and the table owner
*/
requirePermission(Permission.Action.ADMIN);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine
[ https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255345#comment-13255345 ] Andrew Purtell commented on HBASE-5732: --- The reason the User abstraction exists is because the UserGroupInformation API is inconsistent between Hadoop versions. Has this patch been tested on all of 1.0, 0.23/2.0, 0.22 etc.? Remove the SecureRPCEngine and merge the security-related logic in the core engine -- Key: HBASE-5732 URL: https://issues.apache.org/jira/browse/HBASE-5732 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Attachments: rpcengine-merge.patch Remove the SecureRPCEngine and merge the security-related logic in the core engine. Follow up to HBASE-5727. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine
[ https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255398#comment-13255398 ] Andrew Purtell commented on HBASE-5732: --- bq. Andrew, this patch hasn't been tested yet. But if it is made to run with hadoop-1.0, I am pretty sure that it will run on all the versions post 1.0. The UGI class of Hadoop hasn't changed incompatibly since 1.0. I'm all for removing shims and hacks wherever possible, but if we end up here again then this just churns our API along with Hadoop core. Remove the SecureRPCEngine and merge the security-related logic in the core engine -- Key: HBASE-5732 URL: https://issues.apache.org/jira/browse/HBASE-5732 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Attachments: rpcengine-merge.patch Remove the SecureRPCEngine and merge the security-related logic in the core engine. Follow up to HBASE-5727. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5807) TestHLogRecordReader does not compile against Hadoop 2
[
https://issues.apache.org/jira/browse/HBASE-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256123#comment-13256123
]
Andrew Purtell commented on HBASE-5807:
---
JobContext and TaskAttemptContext are only interfaces in 0.23+.
TestHLogRecordReader should be reimplemented with a different approach.
TestHLogRecordReader does not compile against Hadoop 2
--
Key: HBASE-5807
URL: https://issues.apache.org/jira/browse/HBASE-5807
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0, 0.96.0
Reporter: Andrew Purtell
See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-23/144/console
I see this trying to compile against branch 2 also.
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure: Compilation
failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-23/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:[1333,35]
cannot find symbol
[ERROR] symbol : method getJobTracker()
[ERROR] location: class
org.apache.hadoop.mapred.MiniMRCluster.JobTrackerRunner
[ERROR]
[ERROR]
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-23/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[147,46]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-23/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[153,29]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-23/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[194,46]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-23/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[206,29]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-23/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[213,29]
org.apache.hadoop.mapreduce.JobContext is abstract; cannot be instantiated
[ERROR]
[ERROR]
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-23/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java:[226,29]
org.apache.hadoop.mapreduce.TaskAttemptContext is abstract; cannot be
instantiated
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1936) ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services
[ https://issues.apache.org/jira/browse/HBASE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253828#comment-13253828 ] Andrew Purtell commented on HBASE-1936: --- bq. I have checked the code in coprocessor, it can't support the dynamic class during running, though I can refer to the code of how to load jars from HDFS. Can you rephrase this? Ideally we should have one classloader that can load from HDFS, not more than one only slightly different from each other. ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services --- Key: HBASE-1936 URL: https://issues.apache.org/jira/browse/HBASE-1936 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Jieshan Bean Labels: noob Attachments: HBASE-1936-trunk(forReview).patch, cp_from_hdfs.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250013#comment-13250013 ] Andrew Purtell commented on HBASE-5727: --- Removing getConf() was an action that had implications for all coprocessors; AccessController just exposed it by being the only thing in tree currently that uses it. We can unify the build by integrating RPC security in the 0.96 RPC singularity. These issues with secure versus nonsecure build should be as short a transitional phase as possible, i.e. 0.92 and 0.94 use it to bridge Hadoop core differences until 0.96 specifies Hadoop 1.0 or equivalent security shims as a minimum requirement. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250037#comment-13250037 ] Andrew Purtell commented on HBASE-5727: --- bq. If your coprocessor uses the HRegionServer.conf, it should work the same as it always has. If your corprocessor uses HRegion.conf, a compilation error is better than a subtle runtime error that you have to spend a week hunting down. Yes I agree. I didn't mean to imply it was an incorrect change in some way, just that the change had implications (obviously...) bq. Taking a look at the security code, it looks like you want to use HRegionServer.conf because both calling classes seem to access global security data. Yes. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247469#comment-13247469 ] Andrew Purtell commented on HBASE-5727: --- I commented over on HBASE-5451: Subclassing of RPC engines was introduced to deal with incompatibilities in security related classes between Hadoop versions. If we are specifying Hadoop 1.0 (or equivalent shims) as a build requirement, then we could instead use a single RPC envelope and the User abstraction to paper over the remaining differences. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247474#comment-13247474 ] Andrew Purtell commented on HBASE-5727: --- So, to be clear, I mean as part of this work consolidate the RPC engines assuming Hadoop 1.0 or equivalent shims. Then the -P security profile would just build coprocessors. There would be little (no?) need for a security profile at all. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247507#comment-13247507 ] Andrew Purtell commented on HBASE-5727: --- @Stack, @Devaraj: Do you want to try removing the need for separate RPC engines (maybe as a follow on issue)? secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Attachments: 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247517#comment-13247517 ] Andrew Purtell commented on HBASE-5727: --- @Stack. Fixing the build as an interim step toward merging RPC is fine. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Attachments: 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247520#comment-13247520 ] Andrew Purtell commented on HBASE-5727: --- bq. Pushing through Hudson. Let me know if I should run any other test on this patch HudsonQA doesn't test the security profile IIRC. How about running 'mvn -P security -P localTests test' locally? secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Attachments: 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247611#comment-13247611 ] Andrew Purtell commented on HBASE-5451: --- We agreed a simple fix for the build in HBASE-5727 is fine as an interim step. Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 5305v7.txt, 5305v7.txt, rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2, rpc-proto.r5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5706) Dropping fs latency stats since buffer is full spam
[
https://issues.apache.org/jira/browse/HBASE-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245770#comment-13245770
]
Andrew Purtell commented on HBASE-5706:
---
Interesting, this came up today here too. I've been playing around with a
slightly modified version of this patch. Under high load the FS latency stat
buffers can fill up before the metrics thread drains them, doesn't seem an
exceptional condition. Removing logDroppedLatencyStat() and logging around it
gets rid of a number of branches and a CAS from a hot code path, so that's what
we did.
Dropping fs latency stats since buffer is full spam
-
Key: HBASE-5706
URL: https://issues.apache.org/jira/browse/HBASE-5706
Project: HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Assignee: Shaneal Manek
Priority: Minor
Fix For: 0.94.0, 0.96.0
I see tons of this while running tests (note that it's a WARN):
{noformat}
2012-04-03 18:54:47,172 WARN org.apache.hadoop.hbase.io.hfile.HFile: Dropping
fs latency stats since buffer is full
{noformat}
While the code says this:
{noformat}
// we don't want to fill up the logs with this message, so only log it
// once every 30 seconds at most
// I also want to avoid locks on the 'critical path' (the common case will
be
// uncontended) - hence the CAS
private static void logDroppedLatencyStat() {
{noformat}
It doesn't seem like this message is actionnable and even though it's printed
only every 30 seconds it's still very spammy.
We should get rid of it or make it more useful (I don't know which).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5706) Dropping fs latency stats since buffer is full spam
[
https://issues.apache.org/jira/browse/HBASE-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245775#comment-13245775
]
Andrew Purtell commented on HBASE-5706:
---
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via
Tom White)
Dropping fs latency stats since buffer is full spam
-
Key: HBASE-5706
URL: https://issues.apache.org/jira/browse/HBASE-5706
Project: HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Assignee: Shaneal Manek
Priority: Minor
Fix For: 0.94.0, 0.96.0
I see tons of this while running tests (note that it's a WARN):
{noformat}
2012-04-03 18:54:47,172 WARN org.apache.hadoop.hbase.io.hfile.HFile: Dropping
fs latency stats since buffer is full
{noformat}
While the code says this:
{noformat}
// we don't want to fill up the logs with this message, so only log it
// once every 30 seconds at most
// I also want to avoid locks on the 'critical path' (the common case will
be
// uncontended) - hence the CAS
private static void logDroppedLatencyStat() {
{noformat}
It doesn't seem like this message is actionnable and even though it's printed
only every 30 seconds it's still very spammy.
We should get rid of it or make it more useful (I don't know which).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status
[
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241531#comment-13241531
]
Andrew Purtell commented on HBASE-5675:
---
We saw this CME in our internal distro but solved it by bringing forward
TaskMonitor to the latest in 0.92.
Create table fails if we keep refreshing master's UI for task monitor status
Key: HBASE-5675
URL: https://issues.apache.org/jira/browse/HBASE-5675
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4, 0.92.0, 0.94.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
Labels: noob
I tried to create a table with 2K pre-split regions, region assignment was in
middle and i was keep refreshing master's web UI to find the status of the
task using task monitor, table creation was failed and {{META}} was showing
2K regions with server location value is {{null}} and regions weren't
deployed onto region-servers.
{code}
table_ACreating table table_A
java.io.IOException: java.io.IOException:
java.util.ConcurrentModificationException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
at com.test.tools.hbase.schema.main(schema.java:627)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.util.ConcurrentModificationException
at java.util.SubList.checkForComodification(AbstractList.java:752)
at java.util.SubList.add(AbstractList.java:632)
at java.util.SubList.add(AbstractList.java:633)
at java.util.SubList.add(AbstractList.java:633)
..
..
at java.util.SubList.add(AbstractList.java:633)
at java.util.AbstractList.add(AbstractList.java:91)
at
org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.createTable(Unknown Source)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5641) decayingSampleTick1 prevents HBase from shutting down.
[
https://issues.apache.org/jira/browse/HBASE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239634#comment-13239634
]
Andrew Purtell commented on HBASE-5641:
---
+1 on patch but is there a clear case where a utility factory like this should
be producing non-daemon threads?
decayingSampleTick1 prevents HBase from shutting down.
--
Key: HBASE-5641
URL: https://issues.apache.org/jira/browse/HBASE-5641
Project: HBase
Issue Type: Sub-task
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Blocker
Fix For: 0.94.0, 0.96.0
Attachments: 5641.txt
I think this is the problem. It creates a non-daemon thread.
{code}
private static final ScheduledExecutorService TICK_SERVICE =
Executors.newScheduledThreadPool(1,
Threads.getNamedThreadFactory(decayingSampleTick));
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239675#comment-13239675 ] Andrew Purtell commented on HBASE-5617: --- -1 on patch, API changes exceed the scope of this issue as defined. Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5607) Implement scanner caching throttling to prevent too big responses
[ https://issues.apache.org/jira/browse/HBASE-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239684#comment-13239684 ] Andrew Purtell commented on HBASE-5607: --- bq. This issue is for creating an option that throttles a scan regardless of how scanner caching is configured (either number of rows or bytes size). Wouldn't HBASE-2214 be an effective throttle for when the scanner may cross a fat row? What is the rationale for doing more/otherwise? I don't see it clearly spelled out here. Implement scanner caching throttling to prevent too big responses -- Key: HBASE-5607 URL: https://issues.apache.org/jira/browse/HBASE-5607 Project: HBase Issue Type: Improvement Reporter: Ferdy Galema When a misconfigured client retrieves fat rows with a scanner caching value set too high, there is a big chance the regionserver cannot handle the response buffers. (See log example below). Also see the mailing list thread: http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/24819 This issue is for tracking a solution that throttles the scanner caching value in the case the response buffers are too big. A few possible solutions: a) Is a response (repeatedly) over 100MB (configurable), then reduce the scanner-caching by half its size. (In either server or client). b) Introduce a property that defines a fixed (target) response size, instead of defining the numbers of rows to cache. 2012-03-20 07:57:40,092 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020, responseTooLarge for: next(4438820558358059204, 1000) from 172.23.122.15:50218: Size: 105.0m 2012-03-20 07:57:53,226 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020, responseTooLarge for: next(-7429189123174849941, 1000) from 172.23.122.15:50218: Size: 214.4m 2012-03-20 07:57:57,839 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020, responseTooLarge for: next(-7429189123174849941, 1000) from 172.23.122.15:50218: Size: 103.2m 2012-03-20 07:57:59,442 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020, responseTooLarge for: next(-7429189123174849941, 1000) from 172.23.122.15:50218: Size: 101.8m 2012-03-20 07:58:20,025 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60020, responseTooLarge for: next(9033159548564260857, 1000) from 172.23.122.15:50218: Size: 107.2m 2012-03-20 07:58:27,273 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020, responseTooLarge for: next(9033159548564260857, 1000) from 172.23.122.15:50218: Size: 100.1m 2012-03-20 07:58:52,783 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020, responseTooLarge for: next(-8611621895979000997, 1000) from 172.23.122.15:50218: Size: 101.7m 2012-03-20 07:59:02,541 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020, responseTooLarge for: next(-511305750191148153, 1000) from 172.23.122.15:50218: Size: 120.9m 2012-03-20 07:59:25,346 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60020, responseTooLarge for: next(1570572538285935733, 1000) from 172.23.122.15:50218: Size: 107.8m 2012-03-20 07:59:46,805 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020, responseTooLarge for: next(-727080724379055435, 1000) from 172.23.122.15:50218: Size: 102.7m 2012-03-20 08:00:00,138 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020, responseTooLarge for: next(-3701270248575643714, 1000) from 172.23.122.15:50218: Size: 122.1m 2012-03-20 08:00:21,232 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60020, responseTooLarge for: next(5831907615409186602, 1000) from 172.23.122.15:50218: Size: 157.5m 2012-03-20 08:00:23,199 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020, responseTooLarge for: next(5831907615409186602, 1000) from 172.23.122.15:50218: Size: 160.7m 2012-03-20 08:00:28,174 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020, responseTooLarge for: next(5831907615409186602, 1000) from 172.23.122.15:50218: Size: 160.8m 2012-03-20 08:00:32,643 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60020, responseTooLarge for: next(5831907615409186602, 1000) from 172.23.122.15:50218: Size: 182.4m 2012-03-20 08:00:36,826 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020, responseTooLarge for: next(5831907615409186602, 1000) from 172.23.122.15:50218: Size: 237.2m 2012-03-20 08:00:40,850 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020, responseTooLarge for: next(5831907615409186602, 1000) from 172.23.122.15:50218: Size: 212.7m 2012-03-20 08:00:44,736 WARN
[jira] [Commented] (HBASE-5433) [REST] Add metrics to keep track of success/failure count
[
https://issues.apache.org/jira/browse/HBASE-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234792#comment-13234792
]
Andrew Purtell commented on HBASE-5433:
---
I'd say go ahead Lars.
[REST] Add metrics to keep track of success/failure count
-
Key: HBASE-5433
URL: https://issues.apache.org/jira/browse/HBASE-5433
Project: HBase
Issue Type: Improvement
Components: metrics, rest
Affects Versions: 0.94.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
Labels: noob
Fix For: 0.94.0
Attachments: HBASE-5433.trunk.v1.patch
In a production environment, the visibility of successful REST request(s) are
not getting exposed to metric system as we have only one metric (requests)
today.
Proposing to add more metrics such as successful_get_count, failed_get_count,
successful_put_count, failed_put_count
The current implementation increases the request count at the beginning of
the method implementation and it is very hard to monitor requests (unless
turn on debug, find the row_key and validate it in get/scan using hbase
shell), it will be very useful to ops to keep an eye as requests from cross
data-centers are trying to write data to one cluster using REST gateway
through load balancer (and there is no visibility of which REST-server/RS
failed to write data)
{code}
Response update(final CellSetModel model, final boolean replace) {
// for requests
servlet.getMetrics().incrementRequests(1);
..
..
table.put(puts);
table.flushCommits();
ResponseBuilder response = Response.ok();
// for successful_get_count
servlet.getMetrics().incrementSuccessfulGetRequests(1);
return response.build();
} catch (IOException e) {
// for failed_get_count
servlet.getMetrics().incrementFailedGetRequests(1);
throw new WebApplicationException(e,
Response.Status.SERVICE_UNAVAILABLE);
} finally {
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5584) Coprocessor hooks can be called in the respective handlers
[ https://issues.apache.org/jira/browse/HBASE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234806#comment-13234806 ] Andrew Purtell commented on HBASE-5584: --- In the patch I don't see where the unit test waits sufficiently for the async operation to complete. It's going to be racy. Use a CountDownLatch or similar to wait in the test for the handler? Coprocessor hooks can be called in the respective handlers -- Key: HBASE-5584 URL: https://issues.apache.org/jira/browse/HBASE-5584 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5584-1.patch, HBASE-5584.patch Following points can be changed w.r.t to coprocessors - Call preCreate, postCreate, preEnable, postEnable, etc. in their respective handlers - Currently it is called in the HMaster thus making the postApis async w.r.t the handlers - Similar is the case with the balancer. with current behaviour once we are in the postEnable(for eg) we any way need to wait for the main enable handler to be completed. We should ensure that we dont wait in the main thread so again we need to spawn a thread and wait on that. On the other hand if the pre and post api is called on the handlers then only that handler thread will be used in the pre/post apis If the above said plan is ok i can prepare a patch for all such related changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5600) Make Endpoint Coprocessors Available from Thrift
[ https://issues.apache.org/jira/browse/HBASE-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13233058#comment-13233058 ] Andrew Purtell commented on HBASE-5600: --- bq. We could create a thrift method to take the name of the class, method, and an array of params and then call coprocessorExec. This sounds like a reasonable short term thing to do. For now with the dynamic behaviors of the current HRPC based stack we can mostly get away with using the same Java client tools for flexible remote method invocation of Endpoints as with the core interfaces. In the future the fact every Endpoint is really its own little protocol may be more exposed. In this world, the interface passes a blob. Such blobs could recursively contain protobuf (or Thrift) encoding. If going forward we will support Thrift and protobuf (new RPC) clients both, then maybe we can expect server side code will translate from Thrift and protobuf message representations to some common representation, POJO or whatever. In other words, rehydrate from message representation into real classes (via reflection?) At least for Java, protobufs documentation recommends the objects built by the protobuf unmarshaller not be used directly as application classes. I think Thrift has the same practice. So on the server side that might not be so bad. On the client side, given the static nature of Thrift and protobuf message schemas (compiled from IDL) we can't dynamically create messages, so there's no way to hide behind for example a remote invocation proxy or some message builder. It could be different if we used Avro or some other option which can create message schemas at runtime and use those dynamically generated schemas server side. Make Endpoint Coprocessors Available from Thrift Key: HBASE-5600 URL: https://issues.apache.org/jira/browse/HBASE-5600 Project: HBase Issue Type: Improvement Components: thrift Reporter: Ben West Priority: Minor Labels: thrift Currently, the only way to access an endpoint coprocessor via thrift is to modify the schema and Thrift server for every coprocessor function. This is annoying. It should be possible to use your coprocessors without having to mangle HBase core code (since that's the point of coprocessors). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5584) Coprocessor hooks can be called in the respective handlers
[ https://issues.apache.org/jira/browse/HBASE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13230385#comment-13230385 ] Andrew Purtell commented on HBASE-5584: --- Ram, What Mingjie said. The coprocessor hooks can't be asynchronous with respect to client action. Would adding additional hooks for handler actions work for you here? Coprocessor hooks can be called in the respective handlers -- Key: HBASE-5584 URL: https://issues.apache.org/jira/browse/HBASE-5584 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Following points can be changed w.r.t to coprocessors - Call preCreate, postCreate, preEnable, postEnable, etc. in their respective handlers - Currently it is called in the HMaster thus making the postApis async w.r.t the handlers - Similar is the case with the balancer. with current behaviour once we are in the postEnable(for eg) we any way need to wait for the main enable handler to be completed. We should ensure that we dont wait in the main thread so again we need to spawn a thread and wait on that. On the other hand if the pre and post api is called on the handlers then only that handler thread will be used in the pre/post apis If the above said plan is ok i can prepare a patch for all such related changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5584) Coprocessor hooks can be called in the respective handlers
[ https://issues.apache.org/jira/browse/HBASE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13230894#comment-13230894 ] Andrew Purtell commented on HBASE-5584: --- Yes Ram that sounds good. Please consider making a note in the javadoc of MasterObserver which hooks are called synchronously with respect to RPC actions and which are not. Coprocessor hooks can be called in the respective handlers -- Key: HBASE-5584 URL: https://issues.apache.org/jira/browse/HBASE-5584 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Following points can be changed w.r.t to coprocessors - Call preCreate, postCreate, preEnable, postEnable, etc. in their respective handlers - Currently it is called in the HMaster thus making the postApis async w.r.t the handlers - Similar is the case with the balancer. with current behaviour once we are in the postEnable(for eg) we any way need to wait for the main enable handler to be completed. We should ensure that we dont wait in the main thread so again we need to spawn a thread and wait on that. On the other hand if the pre and post api is called on the handlers then only that handler thread will be used in the pre/post apis If the above said plan is ok i can prepare a patch for all such related changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API
[ https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221701#comment-13221701 ] Andrew Purtell commented on HBASE-5371: --- +1 apologies for the delay Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API Key: HBASE-5371 URL: https://issues.apache.org/jira/browse/HBASE-5371 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.92.1, 0.94.0 Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.94.0 Attachments: HBASE-5371-addendum_v1.patch, HBASE-5371_v2.patch, HBASE-5371_v3-noprefix.patch, HBASE-5371_v3.patch We need to introduce something like AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so that clients can check access rights before carrying out the operations. We need this kind of operation for HCATALOG-245, which introduces authorization providers for hbase over hcat. We cannot use getUserPermissions() since it requires ADMIN permissions on the global/table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5500) [Coprocessors] Add hooks for bulk loading actions
[
https://issues.apache.org/jira/browse/HBASE-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220396#comment-13220396
]
Andrew Purtell commented on HBASE-5500:
---
A relevant question from Francis Liu on HBASE-5498:
{quote}
Andrew, thanks for point that discussion out. Can't those two hooks be combined
into one? The user can just ignore the scanner if he doesn't need it. Or is
there a large overhead on even just creating the scanner?
{quote}
There would be some overhead involved with scanning the files. This is
necessary if the CP wants to rewrite or filter some KVs before the HFiles are
installed in the region but otherwise I think the cost can be avoided.
[Coprocessors] Add hooks for bulk loading actions
-
Key: HBASE-5500
URL: https://issues.apache.org/jira/browse/HBASE-5500
Project: HBase
Issue Type: Improvement
Components: coprocessors
Reporter: Andrew Purtell
The API gap for bulk HFile loading was discussed on the mailing list but it
didn't make it into a JIRA. It also came up on HBASE-5498.
See http://search-hadoop.com/m/eEUHK1s4fo81/bulk+loading+and+RegionObservers
The salient detail:
{quote}
A simple and straightforward course of action is to give the CP the
option of rewriting the submitted store file(s) before the regionserver
attempts to validate and move them into the store. This is similar to how CPs
are hooked into compaction: CPs hook compaction by allowing one to wrap the
scanner that is iterating over the store files. So the wrapper gets a chance
to examine the KeyValues being processed and also has an opportunity to
modify or drop them.
Similarly for incoming HFiles for bulk load, the CP could be given a
scanner iterating over those files, if you had a RegionObserver installed.
You would be given the option in effect to rewrite the incoming HFiles before
they are handed over to the RegionServer for addition to the region.
{quote}
I think this is a reasonable approach to interface design, because the fact
you are given a scanner highlights the bulk nature of the input. However I
think there should be two hooks here: one that allows for a simple yes/no
answer as to whether the bulk load should proceed; and one that allows for a
more expensive filtering or transformation or whatever via scanner-like
interface. Bulk loads could be potentially very large so requiring a scan
over them always is not a good idea.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5491) Remove HBaseConfiguration.create() call from coprocessor.Exec class
[ https://issues.apache.org/jira/browse/HBASE-5491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219470#comment-13219470 ] Andrew Purtell commented on HBASE-5491: --- +1 Remove HBaseConfiguration.create() call from coprocessor.Exec class --- Key: HBASE-5491 URL: https://issues.apache.org/jira/browse/HBASE-5491 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.0 Environment: all Reporter: honghua zhu Fix For: 0.92.1 Attachments: HBASE-5491.patch Exec class has a field: private Configuration conf = HBaseConfiguration.create() Client side generates an Exec instance of the class, each initiated Statistics request by ExecRPCInvoker Is so HBaseConfiguration.create for each request needs to call When the server side deserialize the Exec Called once HBaseConfiguration.create in, HBaseConfiguration.create is a time consuming operation. private Configuration conf = HBaseConfiguration.create(); This code is only useful for testing code (org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint.testExecDeserialization), other places with the Exec class, pass a Configuration come, so no need to conf field a default value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[
https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219838#comment-13219838
]
Andrew Purtell commented on HBASE-5498:
---
The API gap was discussed on the mailing list but it didn't make it into a JIRA.
See http://search-hadoop.com/m/eEUHK1s4fo81/bulk+loading+and+RegionObservers
The salient detail:
{quote}
A simple and straightforward course of action is to give the CP the option of
rewriting the submitted store file(s) before the regionserver attempts to
validate and move them into the store. This is similar to how CPs are hooked
into compaction: CPs hook compaction by allowing one to wrap the scanner that
is iterating over the store files. So the wrapper gets a chance to examine the
KeyValues being processed and also has an opportunity to modify or drop them.
Similarly for incoming HFiles for bulk load, the CP could be given a scanner
iterating over those files, if you had a RegionObserver installed. You would be
given the option in effect to rewrite the incoming HFiles before they are
handed over to the RegionServer for addition to the region.
{quote}
I think this is a reasonable approach to interface design, because the fact you
are given a scanner highlights the bulk nature of the input. However I think
there should be two hooks here: one that allows for a simple yes/no answer as
to whether the bulk load should proceed; and one that allows for a more
expensive filtering or transformation or whatever via scanner-like interface.
Bulk loads could be potentially very large so requiring a scan over them always
is not a good idea.
Transferring ownership at the HDFS level can be done as suggested with a
'chown' enhancement IMO.
Secure Bulk Load
Key: HBASE-5498
URL: https://issues.apache.org/jira/browse/HBASE-5498
Project: HBase
Issue Type: Improvement
Reporter: Francis Liu
Design doc:
https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load
Short summary:
Security as it stands does not cover the bulkLoadHFiles() feature. Users
calling this method will bypass ACLs. Also loading is made more cumbersome in
a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data
from user's directory to the hbase directory, which would require certain
write access privileges set.
Our solution is to create a coprocessor which makes use of AuthManager to
verify if a user has write access to the table. If so, launches a MR job as
the hbase user to do the importing (ie rewrite from text to hfiles). One
tricky part this job will have to do is impersonate the calling user when
reading the input files. We can do this by expecting the user to pass an hdfs
delegation token as part of the secureBulkLoad() coprocessor call and extend
an inputformat to make use of that token. The output is written to a
temporary directory accessible only by hbase and then bulkloadHFiles() is
called.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[
https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219878#comment-13219878
]
Andrew Purtell commented on HBASE-5498:
---
{quote}
{quote}
+1
That stop-gap is what we do in production.
Secure Bulk Load
Key: HBASE-5498
URL: https://issues.apache.org/jira/browse/HBASE-5498
Project: HBase
Issue Type: Improvement
Reporter: Francis Liu
Design doc:
https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load
Short summary:
Security as it stands does not cover the bulkLoadHFiles() feature. Users
calling this method will bypass ACLs. Also loading is made more cumbersome in
a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data
from user's directory to the hbase directory, which would require certain
write access privileges set.
Our solution is to create a coprocessor which makes use of AuthManager to
verify if a user has write access to the table. If so, launches a MR job as
the hbase user to do the importing (ie rewrite from text to hfiles). One
tricky part this job will have to do is impersonate the calling user when
reading the input files. We can do this by expecting the user to pass an hdfs
delegation token as part of the secureBulkLoad() coprocessor call and extend
an inputformat to make use of that token. The output is written to a
temporary directory accessible only by hbase and then bulkloadHFiles() is
called.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5483) Allow configurable host to bind to for starting REST server from commandline
[ https://issues.apache.org/jira/browse/HBASE-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13217309#comment-13217309 ] Andrew Purtell commented on HBASE-5483: --- +1 (late) Allow configurable host to bind to for starting REST server from commandline Key: HBASE-5483 URL: https://issues.apache.org/jira/browse/HBASE-5483 Project: HBase Issue Type: Improvement Components: rest Affects Versions: 0.92.0 Reporter: Dan Rosher Priority: Minor Fix For: 0.92.1, 0.94.0 Attachments: HBASE-5483 Current implementation binds to all interfaces, use config to store hbase.rest.host and hbase.rest.port -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5480) Fixups to MultithreadedTableMapper for Hadoop 0.23.2+
[ https://issues.apache.org/jira/browse/HBASE-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13216957#comment-13216957 ] Andrew Purtell commented on HBASE-5480: --- There was a constructor called at that site already, and another constructor called by reflection already above it. This only adds a small incremental cost. Fixups to MultithreadedTableMapper for Hadoop 0.23.2+ - Key: HBASE-5480 URL: https://issues.apache.org/jira/browse/HBASE-5480 Project: HBase Issue Type: Bug Components: mapreduce Reporter: Andrew Purtell Priority: Critical Attachments: HBASE-5480.patch There are two issues: - StatusReporter has a new method getProgress() - Mapper and reducer context objects can no longer be directly instantiated. See attached patch. I'm not thrilled with the added reflection but it was the minimally intrusive change. Raised the priority to critical because compilation fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5433) [REST] Add metrics to keep track of success/failure count
[
https://issues.apache.org/jira/browse/HBASE-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213818#comment-13213818
]
Andrew Purtell commented on HBASE-5433:
---
+1 on the patch. Hudson bot failures seem unrelated.
[REST] Add metrics to keep track of success/failure count
-
Key: HBASE-5433
URL: https://issues.apache.org/jira/browse/HBASE-5433
Project: HBase
Issue Type: Improvement
Components: metrics, rest
Affects Versions: 0.94.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
Labels: noob
Fix For: 0.94.0
Attachments: HBASE-5433.trunk.v1.patch
In a production environment, the visibility of successful REST request(s) are
not getting exposed to metric system as we have only one metric (requests)
today.
Proposing to add more metrics such as successful_get_count, failed_get_count,
successful_put_count, failed_put_count
The current implementation increases the request count at the beginning of
the method implementation and it is very hard to monitor requests (unless
turn on debug, find the row_key and validate it in get/scan using hbase
shell), it will be very useful to ops to keep an eye as requests from cross
data-centers are trying to write data to one cluster using REST gateway
through load balancer (and there is no visibility of which REST-server/RS
failed to write data)
{code}
Response update(final CellSetModel model, final boolean replace) {
// for requests
servlet.getMetrics().incrementRequests(1);
..
..
table.put(puts);
table.flushCommits();
ResponseBuilder response = Response.ok();
// for successful_get_count
servlet.getMetrics().incrementSuccessfulGetRequests(1);
return response.build();
} catch (IOException e) {
// for failed_get_count
servlet.getMetrics().incrementFailedGetRequests(1);
throw new WebApplicationException(e,
Response.Status.SERVICE_UNAVAILABLE);
} finally {
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5434) [REST] Include more metrics in cluster status request
[
https://issues.apache.org/jira/browse/HBASE-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214041#comment-13214041
]
Andrew Purtell commented on HBASE-5434:
---
+1
Looks good Mubarak.
[REST] Include more metrics in cluster status request
-
Key: HBASE-5434
URL: https://issues.apache.org/jira/browse/HBASE-5434
Project: HBase
Issue Type: Improvement
Components: metrics, rest
Affects Versions: 0.94.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
Priority: Minor
Labels: noob
Fix For: 0.94.0
Attachments: HBASE-5434.trunk.v1.patch
/status/cluster shows only
{code}
stores=2
storefiless=0
storefileSizeMB=0
memstoreSizeMB=0
storefileIndexSizeMB=0
{code}
for a region but master web-ui shows
{code}
stores=1,
storefiles=0,
storefileUncompressedSizeMB=0
storefileSizeMB=0
memstoreSizeMB=0
storefileIndexSizeMB=0
readRequestsCount=0
writeRequestsCount=0
rootIndexSizeKB=0
totalStaticIndexSizeKB=0
totalStaticBloomSizeKB=0
totalCompactingKVs=0
currentCompactedKVs=0
compactionProgressPct=NaN
{code}
In a write-heavy REST gateway based production environment, ops team needs to
verify whether write counters are getting incremented per region (they do run
/status/cluster on each REST server), we can get the same values from
*rpc.metrics.put_num_ops* and *hbase.regionserver.writeRequestsCount* but
some home-grown tools needs to parse the output of /status/cluster and
updates the dashboard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API
[
https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13211033#comment-13211033
]
Andrew Purtell commented on HBASE-5371:
---
{quote}
bq. It can't because it changes the protocol version of
AccessControllerProtocol.
Right, since we bumped the protocol version, it will be incompatible. [...]
What about re-changing the version to 1, since we just added a new method, but
not changed anything on the wire, it should be compatible. The only catch is
that if you invoke the new API from a new client, but the server is using the
old version, you would get a NoSuchMethod or smt.
{quote}
I've seen this approach used in HDFS. (At least in CDH.) The client can catch
the NoSuchMethodException, set a boolean or similar to note that it is talking
with an older version, and try an alternate strategy.
I think this is a reasonable approach until we have a more general solution for
cross-version (and backwards) RPC compatibility.
Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons)
API
Key: HBASE-5371
URL: https://issues.apache.org/jira/browse/HBASE-5371
Project: HBase
Issue Type: Sub-task
Components: security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.94.0
Attachments: HBASE-5371_v2.patch, HBASE-5371_v3-noprefix.patch,
HBASE-5371_v3.patch
We need to introduce something like
AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so
that clients can check access rights before carrying out the operations. We
need this kind of operation for HCATALOG-245, which introduces authorization
providers for hbase over hcat. We cannot use getUserPermissions() since it
requires ADMIN permissions on the global/table level.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API
[ https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13211035#comment-13211035 ] Andrew Purtell commented on HBASE-5371: --- @Enis Make sure you add the new methods at the bottom of the interface definition. Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API Key: HBASE-5371 URL: https://issues.apache.org/jira/browse/HBASE-5371 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.92.1 Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.94.0 Attachments: HBASE-5371_v2.patch, HBASE-5371_v3-noprefix.patch, HBASE-5371_v3.patch We need to introduce something like AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so that clients can check access rights before carrying out the operations. We need this kind of operation for HCATALOG-245, which introduces authorization providers for hbase over hcat. We cannot use getUserPermissions() since it requires ADMIN permissions on the global/table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5434) [REST] Include more metrics in cluster status request
[
https://issues.apache.org/jira/browse/HBASE-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13211222#comment-13211222
]
Andrew Purtell commented on HBASE-5434:
---
+1
Te cluster status representation hasn't been updated since 0.90.
[REST] Include more metrics in cluster status request
-
Key: HBASE-5434
URL: https://issues.apache.org/jira/browse/HBASE-5434
Project: HBase
Issue Type: Improvement
Components: metrics, rest
Affects Versions: 0.94.0
Reporter: Mubarak Seyed
Assignee: Mubarak Seyed
Priority: Minor
Labels: noob
Fix For: 0.94.0
/status/cluster shows only
{code}
stores=2
storefiless=0
storefileSizeMB=0
memstoreSizeMB=0
storefileIndexSizeMB=0
{code}
for a region but master web-ui shows
{code}
stores=1,
storefiles=0,
storefileUncompressedSizeMB=0
storefileSizeMB=0
memstoreSizeMB=0
storefileIndexSizeMB=0
readRequestsCount=0
writeRequestsCount=0
rootIndexSizeKB=0
totalStaticIndexSizeKB=0
totalStaticBloomSizeKB=0
totalCompactingKVs=0
currentCompactedKVs=0
compactionProgressPct=NaN
{code}
In a write-heavy REST gateway based production environment, ops team needs to
verify whether write counters are getting incremented per region (they do run
/status/cluster on each REST server), we can get the same values from
*rpc.metrics.put_num_ops* and *hbase.regionserver.writeRequestsCount* but
some home-grown tools needs to parse the output of /status/cluster and
updates the dashboard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5195) [Coprocessors] preGet hook does not allow overriding or wrapping filter on incoming Get
[ https://issues.apache.org/jira/browse/HBASE-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13210569#comment-13210569 ] Andrew Purtell commented on HBASE-5195: --- +1 for including in 0.92.1 [Coprocessors] preGet hook does not allow overriding or wrapping filter on incoming Get --- Key: HBASE-5195 URL: https://issues.apache.org/jira/browse/HBASE-5195 Project: HBase Issue Type: Bug Affects Versions: 0.94.0, 0.92.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.94.0, 0.92.1 Attachments: HBASE-5195.patch Without the ability to wrap the internal Scan on the Get, we can't override (or protect, in the case of access control) Gets as we can Scans. The result is inconsistent behavior. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5371) Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API
[
https://issues.apache.org/jira/browse/HBASE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13208547#comment-13208547
]
Andrew Purtell commented on HBASE-5371:
---
{quote}
@Andy:
Do you think this should go into 0.92.1 ?
{quote}
It can't because it changes the protocol version of AccessControllerProtocol.
Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons)
API
Key: HBASE-5371
URL: https://issues.apache.org/jira/browse/HBASE-5371
Project: HBase
Issue Type: Sub-task
Components: security
Affects Versions: 0.94.0, 0.92.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Attachments: HBASE-5371_v2.patch, HBASE-5371_v3-noprefix.patch,
HBASE-5371_v3.patch
We need to introduce something like
AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so
that clients can check access rights before carrying out the operations. We
need this kind of operation for HCATALOG-245, which introduces authorization
providers for hbase over hcat. We cannot use getUserPermissions() since it
requires ADMIN permissions on the global/table level.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5355) Compressed RPC's for HBase
[ https://issues.apache.org/jira/browse/HBASE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13205284#comment-13205284 ] Andrew Purtell commented on HBASE-5355: --- bq. A while back I saw someone who had proposed a compressed representation of KV that had 'natural prefix' compression. It took advantage of the fact that KVs are typically stored sorted, so one could have a 'this KV has the same row as the previous' flag, and ditto for columns, etc. I think this was something I mentioned in some random JIRA. Compressed RPC's for HBase -- Key: HBASE-5355 URL: https://issues.apache.org/jira/browse/HBASE-5355 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.89.20100924 Reporter: Karthik Ranganathan Assignee: Karthik Ranganathan Some application need ability to do large batched writes and reads from a remote MR cluster. These eventually get bottlenecked on the network. These results are also pretty compressible sometimes. The aim here is to add the ability to do compressed calls to the server on both the send and receive paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5372) Table mutation operations should check table level rights, not global rights
[ https://issues.apache.org/jira/browse/HBASE-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13205282#comment-13205282 ] Andrew Purtell commented on HBASE-5372: --- bq. However, for this maybe we have to revisit table ownership rights. Currently, the table owner has every right on the table, and this is not managed through the normal grant/revoke operations, but on the table metadata. We may want to remove table ownership, but introduce default table creation rights, which means that when a user creates a table, she automatically get those rights allocated. But another user can grant extra rights, or revoke them. Sure, makes sense. We opted for simplicity in the initial implementation. Table mutation operations should check table level rights, not global rights - Key: HBASE-5372 URL: https://issues.apache.org/jira/browse/HBASE-5372 Project: HBase Issue Type: Sub-task Components: security Reporter: Enis Soztutar Assignee: Enis Soztutar getUserPermissions(tableName)/grant/revoke and drop/modify table operations should not check for global CREATE/ADMIN rights, but table CREATE/ADMIN rights. The reasoning is that if a user is able to admin or read from a table, she should be able to read the table's permissions. We can choose whether we want only READ or ADMIN permissions for getUserPermission(). Since we check for global permissions first for table permissions, configuring table access using global permissions will continue to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5352) ACL improvements
[ https://issues.apache.org/jira/browse/HBASE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13203267#comment-13203267 ] Andrew Purtell commented on HBASE-5352: --- bq. Introduce something like AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so that clients can check access rights before carrying out the operations. Sounds good. bq. getUserPermissions(tableName)/grant/revoke and drop/modify table operations should not check for global CREATE/ADMIN rights, but table CREATE/ADMIN rights. This certainly makes sense for grant/revoke. However, creating, dropping, or modifying a table has global ramifications on the cluster. When making changes here it should remain possible to require global CREATE/ADMIN rights for those actions by policy. This way a site admin can prevent users from taking actions that perturb region assignment if desired. ACL improvements Key: HBASE-5352 URL: https://issues.apache.org/jira/browse/HBASE-5352 Project: HBase Issue Type: Improvement Components: security Affects Versions: 0.94.0, 0.92.1 Reporter: Enis Soztutar Assignee: Enis Soztutar In this issue I would like to open discussion for a few minor ACL related improvements. The proposed changes are as follows: 1. Introduce something like AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so that clients can check access rights before carrying out the operations. We need this kind of operation for HCATALOG-245, which introduces authorization providers for hbase over hcat. We cannot use getUserPermissions() since it requires ADMIN permissions on the global/table level. 2. getUserPermissions(tableName)/grant/revoke and drop/modify table operations should not check for global CREATE/ADMIN rights, but table CREATE/ADMIN rights. The reasoning is that if a user is able to admin or read from a table, she should be able to read the table's permissions. We can choose whether we want only READ or ADMIN permissions for getUserPermission(). Since we check for global permissions first for table permissions, configuring table access using global permissions will continue to work. 3. Grant/Revoke global permissions - HBASE-5342 (included for completeness) From all 3, we may want to backport the first one to 0.92 since without it, Hive/Hcatalog cannot use Hbase's authorization mechanism effectively. I will create subissues and convert HBASE-5342 to a subtask when we get some feedback, and opinions for going further. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5343) Access control API in HBaseAdmin.java
[ https://issues.apache.org/jira/browse/HBASE-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202133#comment-13202133 ] Andrew Purtell commented on HBASE-5343: --- bq. We can add grant/revoke/user_permission commands similar to the shell interface to HBaseAdmin assuming HBASE-5341 is in. -1 This issue should be resolved as 'invalid'. Access control API in HBaseAdmin.java --- Key: HBASE-5343 URL: https://issues.apache.org/jira/browse/HBASE-5343 Project: HBase Issue Type: Improvement Components: client, coprocessors, security Affects Versions: 0.94.0, 0.92.1 Reporter: Enis Soztutar Assignee: Enis Soztutar To use the access control mechanism added in HBASE-3025, users should either use the shell interface, or use the coprocessor API directly, which is not very user friendly. We can add grant/revoke/user_permission commands similar to the shell interface to HBaseAdmin assuming HBASE-5341 is in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5341) HBase build artifact should include security code by defult
[ https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13202135#comment-13202135 ] Andrew Purtell commented on HBASE-5341: --- If we drop support for 0.20.x then compilation issues mostly go away. It won't fail for lack of security APIs in Hadoop. I lean with Gary's view that security belongs in a (Maven) module. HBase build artifact should include security code by defult Key: HBASE-5341 URL: https://issues.apache.org/jira/browse/HBASE-5341 Project: HBase Issue Type: Improvement Components: build, security Affects Versions: 0.94.0, 0.92.1 Reporter: Enis Soztutar Assignee: Enis Soztutar Hbase 0.92.0 was released with two artifacts, plain and security. The security code is built with -Psecurity. There are two tarballs, but only the plain jar in maven repo at repository.a.o. I see no reason to do a separate artifact for the security related code, since 0.92 already depends on secure Hadoop 1.0.0, and all of the security related code is not loaded by default. In this issue, I propose, we merge the code under /security to src/ and remove the maven profile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5306) Add support for protocol buffer based RPC
[ https://issues.apache.org/jira/browse/HBASE-5306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198509#comment-13198509 ] Andrew Purtell commented on HBASE-5306: --- Please consider also the native protobuf support in the REST connector. It should be refactored to use what core provides. Add support for protocol buffer based RPC - Key: HBASE-5306 URL: https://issues.apache.org/jira/browse/HBASE-5306 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Devaraj Das Assignee: Devaraj Das This will help HBase to achieve wire compatibility across versions. The idea (to start with) is to leverage the recent work that has gone in in the Hadoop core in this area. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5284) TestLogRolling.java doesn't compile against the latest 0.23.1-SNAPSHOT
[
https://issues.apache.org/jira/browse/HBASE-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13194086#comment-13194086
]
Andrew Purtell commented on HBASE-5284:
---
This is being addressed as part of HBASE-5212, but I don't think we need to
close this as a dup. Split handling of the two problems into these separate
issues.
TestLogRolling.java doesn't compile against the latest 0.23.1-SNAPSHOT
--
Key: HBASE-5284
URL: https://issues.apache.org/jira/browse/HBASE-5284
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Here's how to reproduce:
{noformat}
$ mvn clean -DskipTests -Dhadoop.profile=23 -Dinstall site assembly:assembly
-Dmaven.repo.local=/home/rvs/.m2/repository
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR]
/home/rvs/src/bigtop/output/hbase/hbase-0.92.0/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[341,33]
cannot find symbol
[ERROR] symbol : variable dnRegistration
[ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
[ERROR] - [Help 1]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13193415#comment-13193415 ] Andrew Purtell commented on HBASE-3025: --- See HBASE-4990. Destined for the site manual. The piece I have left to do is a capture of an example shell session. I have such a capture but it's led to follow on jiras that need to be resolved for 0.92.1 Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch, HBASE-3025_6.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[
https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192307#comment-13192307
]
Andrew Purtell commented on HBASE-4720:
---
Ted, I am going to revert your commit.
{quote}
bq. The URI format for requests is '/table/row/ ...' This violates that by
adding, just for check-and cases, a prefix. Having a special case like that
should be avoided. What about handling this in TableResource, with a query
parameter? '/table/row/?check' E.g.Then you won't need
CheckAndXTableResource classes. Additionally use the appropriate HTTP
operations. PUT/POST for check-and-put. DELETE for check-and-delete. The spec
does not forbid bodies in DELETE requests. (I am unsure if Jetty/Jersey will
support it however.)
We have discussed the design choices earlier (refer comments in the same JIRA),
Stack and Ted have voted for option # 2 (/checkandput, /checkanddelete) option.
If i have to go back to option #1 then i will have to re-work most of the stuff
here.
{quote}
This has not changed, therefore -1.
Implement atomic update operations (checkAndPut, checkAndDelete) for REST
client/server
Key: HBASE-4720
URL: https://issues.apache.org/jira/browse/HBASE-4720
Project: HBase
Issue Type: Improvement
Reporter: Daniel Lord
Assignee: Mubarak Seyed
Fix For: 0.94.0
Attachments: HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch,
HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch,
HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.v1.patch,
HBASE-4720.v3.patch
I have several large application/HBase clusters where an application node
will occasionally need to talk to HBase from a different cluster. In order
to help ensure some of my consistency guarantees I have a sentinel table that
is updated atomically as users interact with the system. This works quite
well for the regular hbase client but the REST client does not implement
the checkAndPut and checkAndDelete operations. This exposes the application
to some race conditions that have to be worked around. It would be ideal if
the same checkAndPut/checkAndDelete operations could be supported by the REST
client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192308#comment-13192308 ] Andrew Purtell commented on HBASE-4720: --- Actually Ted you should now honor my -1 and revert that commit. Next time allow me chance to find out there is even a discussion happening. Stack brought this to my attention late last night only. Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.94.0 Attachments: HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192328#comment-13192328 ] Andrew Purtell commented on HBASE-4720: --- Ted, I raised a objection on this issue and you committed it with that objection unaddressed before I had a chance to ack the commit candidate. I apologize if it was not clear my objection earlier was a -1. It was. I thought it clear enough, my mistake. Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.94.0 Attachments: HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192329#comment-13192329 ] Andrew Purtell commented on HBASE-4720: --- Anyway, my apologies, but this commit must be reverted. It is poor and IMO unacceptable design to have such an inconsistency in how request URLs should be constructed. Either the current semantics for REST paths remain the same, that is /table/row/ ..., or every operation is changed to specify the operation first, e.g. /operation/table/row ... . Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.94.0 Attachments: HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5258) Move coprocessors set out of RegionLoad
[ https://issues.apache.org/jira/browse/HBASE-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191589#comment-13191589 ] Andrew Purtell commented on HBASE-5258: --- bq. This flexibility is causing extra cost in the region server to Master communication and increasing the footprint of Master heap. No doubt it is redundant to have each region report a coprocessor given how the framework currently works: All regions for a table will have an identical set of coprocessors loaded, or there is something bad happening. bq. Would HServerLoad be a better place for this set ? I have no major objection. However, maybe we want a way to know if something bad happened on a region and a coprocessor on it went away? One could comb logs but that is hardly a convenient way to get online state. Move coprocessors set out of RegionLoad --- Key: HBASE-5258 URL: https://issues.apache.org/jira/browse/HBASE-5258 Project: HBase Issue Type: Task Reporter: Zhihong Yu When I worked on HBASE-5256, I revisited the code related to Ser/De of coprocessors set in RegionLoad. I think the rationale for embedding coprocessors set is for maximum flexibility where each region can load different coprocessors. This flexibility is causing extra cost in the region server to Master communication and increasing the footprint of Master heap. Would HServerLoad be a better place for this set ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5258) Move coprocessors set out of RegionLoad
[ https://issues.apache.org/jira/browse/HBASE-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191639#comment-13191639 ] Andrew Purtell commented on HBASE-5258: --- bq. for a mis-behaving coprocessor we either remove the buggy coprocessor ... from the coprocessor host for the given region (in the case of RegionCoprocessorHost) only... Move coprocessors set out of RegionLoad --- Key: HBASE-5258 URL: https://issues.apache.org/jira/browse/HBASE-5258 Project: HBase Issue Type: Task Reporter: Zhihong Yu When I worked on HBASE-5256, I revisited the code related to Ser/De of coprocessors set in RegionLoad. I think the rationale for embedding coprocessors set is for maximum flexibility where each region can load different coprocessors. This flexibility is causing extra cost in the region server to Master communication and increasing the footprint of Master heap. Would HServerLoad be a better place for this set ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5266) Add documentation for ColumnRangeFilter
[ https://issues.apache.org/jira/browse/HBASE-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191699#comment-13191699 ] Andrew Purtell commented on HBASE-5266: --- Maybe you could work up an example of that for the docs Lars? Add documentation for ColumnRangeFilter --- Key: HBASE-5266 URL: https://issues.apache.org/jira/browse/HBASE-5266 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.94.0 There are only a few lines of documentation for ColumnRangeFilter. Given the usefulness of this filter for efficient intra-row scanning (see HASE-5229 and HBASE-4256), we should make this filter more prominent in the documentation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189256#comment-13189256 ] Andrew Purtell commented on HBASE-5228: --- @Ted: I observed the blocking on this code path eliminated in runs under jprofiler after the patch was applied. @Ben: It would be good to confirm this on your side. [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.94.0, 0.92.1, 0.90.6 Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189556#comment-13189556 ] Andrew Purtell commented on HBASE-5228: --- There could be more going on, but let's get this problem settled and keep digging. I'll commit to trunk, 0.92, and 0.90 tonight after the meetup. [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.94.0, 0.92.1, 0.90.6 Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188832#comment-13188832 ] Andrew Purtell commented on HBASE-5228: --- Jack confirmed in private correspondence that this can come out. I'd like to do it ASAP as part of 0.92 (and any further 0.90 release) so we don't pick up a user of this misfeature. [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-5228.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[
https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188955#comment-13188955
]
Andrew Purtell commented on HBASE-5228:
---
Test results for 0.92:
{noformat}
Tests in error:
testMasterFailoverWithMockedRIT(org.apache.hadoop.hbase.master.TestMasterFailover):
test timed out after 18 milliseconds
testRetrying(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor): test
timed out after 18 milliseconds
testGetRegionsCatalogTables(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor):
Failed after attempts=10, exceptions:
testTableExists(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor):
Connection refused
testGetRegion(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor):
Connection refused
testScanMetaForTable(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor):
Failed after attempts=10, exceptions:
{noformat}
Tests run: 1076, Failures: 0, Errors: 1, Skipped: 8
Test results for trunk:
{noformat}
Results :
Tests run: 533, Failures: 0, Errors: 0, Skipped: 2
{noformat}
Test failures for 0.92 are worrisome, but not related to this patch.
[REST] Rip out transform feature
--
Key: HBASE-5228
URL: https://issues.apache.org/jira/browse/HBASE-5228
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0, 0.90.5
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.94.0, 0.92.1, 0.90.6
Attachments: HBASE-5228-0.92.patch, HBASE-5228-trunk.patch
The 'transform' feature, where REST can be instructed, via a table attribute,
to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of
column values before serving them up to a client or storing them into HBase,
was added some time ago at the request of Jack Levin. I have since come to
regret it, it was not a well thought out feature:
- This is really an application concern.
- It adds significant overhead to request processing: Periodically a
HBaseAdmin is used to retrieve the table descriptor, in order to scan through
table attributes for transformation directives.
I think it is best to rip it out, its a real problem area, and REST should be
no more concerned about data formats than the Java API.
I doubt anyone uses this, not even Jack. Will need to follow up with him to
confirm.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5174) Coalesce aborted tasks in the TaskMonitor
[
https://issues.apache.org/jira/browse/HBASE-5174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185187#comment-13185187
]
Andrew Purtell commented on HBASE-5174:
---
bq. Considering ABORTED task would be cleaned up in 1 minute, I wonder if the
complexity introduced is worth it.
On the other hand the display would be cleaner with coalescing, so perhaps
failed or aborted tasks could remain displayed for a longer period of time.
Coalesce aborted tasks in the TaskMonitor
-
Key: HBASE-5174
URL: https://issues.apache.org/jira/browse/HBASE-5174
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Fix For: 0.94.0, 0.92.1
Some tasks can get repeatedly canceled like flushing when splitting is going
on, in the logs it looks like this:
{noformat}
2012-01-10 19:28:29,164 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region
test1,,1326223218996.3eea0d89af7b851c3a9b4246389a4f2c. due to global heap
pressure
2012-01-10 19:28:29,164 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
NOT flushing memstore for region
test1,,1326223218996.3eea0d89af7b851c3a9b4246389a4f2c., flushing=false,
writesEnabled=false
2012-01-10 19:28:29,164 DEBUG
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
because memory above low water=1.6g
2012-01-10 19:28:29,164 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region
test1,,1326223218996.3eea0d89af7b851c3a9b4246389a4f2c. due to global heap
pressure
2012-01-10 19:28:29,164 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
NOT flushing memstore for region
test1,,1326223218996.3eea0d89af7b851c3a9b4246389a4f2c., flushing=false,
writesEnabled=false
2012-01-10 19:28:29,164 DEBUG
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
because memory above low water=1.6g
2012-01-10 19:28:29,164 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region
test1,,1326223218996.3eea0d89af7b851c3a9b4246389a4f2c. due to global heap
pressure
2012-01-10 19:28:29,164 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
NOT flushing memstore for region
test1,,1326223218996.3eea0d89af7b851c3a9b4246389a4f2c., flushing=false,
writesEnabled=false
{noformat}
But in the TaskMonitor UI you'll get MAX_TASKS (1000) displayed on top of the
regions. Basically 1000x:
{noformat}
Tue Jan 10 19:28:29 UTC 2012 Flushing
test1,,1326223218996.3eea0d89af7b851c3a9b4246389a4f2c. ABORTED (since 31sec
ago) Not flushing since writes not enabled (since 31sec ago)
{noformat}
It's ugly and I'm sure some users will freak out seeing this, plus you have
to scroll down all the way to see your regions. Coalescing consecutive
aborted tasks seems like a good solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5164) Better HTable resource consumption in CoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183069#comment-13183069 ] Andrew Purtell commented on HBASE-5164: --- We added getTable to the CP API with that risk in mind. The notion is CPs are not allowed to just instantiate HTables, instead they ask for a more constrained server side IPC. No such implementation yet exists, so getTable essentially is a placeholder. If we consider coprocessors hosted in an external process, but still on the server, the risks are not much lower than in process. A stuck table client in the external CP will hold up the initiator anyway, and in fact this makes external CP handling more problematic than if table clients were simply not allowed server side at all. However I think many useful CP use cases must have a table access scope wider than the single region. What we need is a lightweight server to server IPC? Would present HTableInterface but lack the fat client semantics. Better HTable resource consumption in CoprocessorHost - Key: HBASE-5164 URL: https://issues.apache.org/jira/browse/HBASE-5164 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Lars Hofhansl Priority: Minor Fix For: 0.94.0 HBASE-4805 allows for more control over HTable's resource consumption. This is currently not used by CoprocessorHost (even though it would even be more critical to control this inside the RegionServer). It's not immediate obvious how to do that. Maybe CoprocessorHost should maintain a lazy ExecutorService and HConnection and reuse both for all HTables retrieved via CoprocessorEnvironment.getTable(...). Not sure how critical this is, but I feel without this it is dangerous to use getTable, as it would lead to all resource consumption problems we find in the client, but inside a crucial part of the HBase servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179727#comment-13179727
]
Andrew Purtell commented on HBASE-5097:
---
+1 on fixed patch.
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5097.patch, HBASE-5097_1.patch, HBASE-5097_2.patch
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176786#comment-13176786
]
Andrew Purtell commented on HBASE-5097:
---
bq. We should save the non-null s before calling postScannerOpen(). If
postScannerOpen() returns null, we can use the saved scanner.
... and emit a warning. Sounds good.
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5101) Add a max number of regions per regionserver limit
[ https://issues.apache.org/jira/browse/HBASE-5101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176949#comment-13176949 ] Andrew Purtell commented on HBASE-5101: --- This would disable splitting on the RS at the limit? Add a max number of regions per regionserver limit -- Key: HBASE-5101 URL: https://issues.apache.org/jira/browse/HBASE-5101 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.92.0 Reporter: Jonathan Hsieh In a testing environment, a cluster got to a state with more than 1500 regions per region server, and essentially became stuck and unavailable. We could add a limit to the number of regions that a region server can serve to prevent this from happening. This looks like it could be implemented in the core or as a coprocessor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5097) Coprocessor RegionObserver implementation without preScannerOpen and postScannerOpen Impl is throwing NPE and so failing the system initialization.
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176117#comment-13176117
]
Andrew Purtell commented on HBASE-5097:
---
Are you inheriting from {{BaseRegionObserver}}? I'd guess not?
Coprocessor RegionObserver implementation without preScannerOpen and
postScannerOpen Impl is throwing NPE and so failing the system initialization.
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5097) Coprocessor RegionObserver implementation without preScannerOpen and postScannerOpen Impl is throwing NPE and so failing the system initialization.
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176287#comment-13176287
]
Andrew Purtell commented on HBASE-5097:
---
Wouldn't hurt.
Coprocessor RegionObserver implementation without preScannerOpen and
postScannerOpen Impl is throwing NPE and so failing the system initialization.
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176322#comment-13176322 ] Andrew Purtell commented on HBASE-5061: --- bq. From looking at the code, this would produce data locality with respect to certain host (if I understand this correctly). Yes this is the intent, to report locality to a given regionserver host, by default the local host as determined by a HBase style reverse lookup, assuming ops would run it either on an ops box (or the master) and supply the desired local host name via the '-h' option, or local to the RS. I was thinking one use case could be to iterate over each cluster node and trigger major compactions for region(s) depending on the output of this tool. Of you are looking for a whole cluster report? StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: StoreFileLocalityChecker.java org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176326#comment-13176326 ] Andrew Purtell commented on HBASE-5061: --- So, currently this tool is meant to check the locality of files or regions to a given single host, however it could be modified such that if the '-h' option is supplied then the report looks only at the given host, otherwise it will enumerate the live hosts in ClusterStatus and return results for all. While at it, a new option '-j' for reporting in JSON. Is this more what you have in mind Lars? StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: StoreFileLocalityChecker.java org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176091#comment-13176091 ] Andrew Purtell commented on HBASE-5061: --- @Weidong, yes this uses the utility functions done for HBASE-4114. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: StoreFileLocalityChecker.java org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5089) NIO based REST server
[ https://issues.apache.org/jira/browse/HBASE-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13174983#comment-13174983 ] Andrew Purtell commented on HBASE-5089: --- Cloudera ships a Jetty 6 version with NIO, drop in replacement, open source: https://github.com/toddlipcon/jetty-hadoop-fix I don't have an opinion about what servlet server to use. Once upon a time we had a WAR target. We dropped it due to build difficulties when moving to Maven initially. We could bring it back, leave it up to the user what servlet container to use. NIO based REST server - Key: HBASE-5089 URL: https://issues.apache.org/jira/browse/HBASE-5089 Project: HBase Issue Type: Improvement Components: rest Affects Versions: 0.94.0 Reporter: Mubarak Seyed Labels: noob Fix For: 0.94.0 Rest server code uses Jetty 6.1.26, which uses old IO (org.mortbay.jetty.Server). It would be good if Rest server uses NIO based servlet server. The options are as follows: # [Jetty 7 or above (NIO based)|http://download.eclipse.org/jetty/stable-8/xref/] # [Netty-based Rest server|https://github.com/netty/netty/wiki/Related-projects] # [Grizzly|http://grizzly.java.net] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4235) Attempts to reconnect to expired ZooKeeper sessions
[ https://issues.apache.org/jira/browse/HBASE-4235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13174563#comment-13174563 ] Andrew Purtell commented on HBASE-4235: --- @Ken: Not sure if it is the same issue. ZooKeeper logs say the session expired? Pardon, when I spoke with Camille at Hadoop World NYC the verdict was the patch on 1159 was not correct, and this was the basis for 'ClientCnxn does not propagate session expiration indication'. This got jumbled in my head to an issue resolution. I do have the impression that jira will be resolved that way. Attempts to reconnect to expired ZooKeeper sessions --- Key: HBASE-4235 URL: https://issues.apache.org/jira/browse/HBASE-4235 Project: HBase Issue Type: Task Affects Versions: 0.92.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell In a couple of instances of short network outages, we have observed afterward zombie HBase processes attempting over and over to reconnect to expired ZooKeeper sessions. We believe this is due to ZOOKEEPER-1159. Opening this issue as reference to that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173373#comment-13173373 ] Andrew Purtell commented on HBASE-5074: --- +1 support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5040) Secure HBase builds fail
[
https://issues.apache.org/jira/browse/HBASE-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171651#comment-13171651
]
Andrew Purtell commented on HBASE-5040:
---
@Stack: The current patch on this issue works here. Build completes, I can set
up a working all-secure cluster with Hadoop 1.0.0rc2, HBase 0.92.0rc and ZK
3.4.0. But it may not address a similar issue in the 0.22 build that Ted
mentioned. (I haven't checked that.)
Secure HBase builds fail
Key: HBASE-5040
URL: https://issues.apache.org/jira/browse/HBASE-5040
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Fix For: 0.92.0
Attachments: 5040.txt
I saw the following in HBase-0.92-security build #39:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR]
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
[ERROR] - [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
{code}
The above was probably introduced by HBASE-5006
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5062) Missing logons if security is enabled
[ https://issues.apache.org/jira/browse/HBASE-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171733#comment-13171733 ] Andrew Purtell commented on HBASE-5062: --- Deleted invalid HadoopQA results. Missing logons if security is enabled - Key: HBASE-5062 URL: https://issues.apache.org/jira/browse/HBASE-5062 Project: HBase Issue Type: Bug Components: rest, security, thrift Affects Versions: 0.92.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-5062-v2.patch, HBASE-5062.patch Somehow the attached changes are missing from the security integration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5040) Secure HBase builds fail
[
https://issues.apache.org/jira/browse/HBASE-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171133#comment-13171133
]
Andrew Purtell commented on HBASE-5040:
---
Just update the POM to specify Hadoop 1.0.0rc2 in the Hadoop profile instead of
the custom version. That will fix the build issue. HADOOP-7070 is not needed,
see the comments on HADOOP-7853 and HADOOP-7929.
Secure HBase builds fail
Key: HBASE-5040
URL: https://issues.apache.org/jira/browse/HBASE-5040
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Fix For: 0.92.0
I saw the following in HBase-0.92-security build #39:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR]
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
[ERROR] - [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
{code}
The above was probably introduced by HBASE-5006
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-904) Make client capable of riding over a cluster restart
[ https://issues.apache.org/jira/browse/HBASE-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170495#comment-13170495 ] Andrew Purtell commented on HBASE-904: -- Testing the capability of the REST gateway to handle a cluster restart in 0.92. Make client capable of riding over a cluster restart Key: HBASE-904 URL: https://issues.apache.org/jira/browse/HBASE-904 Project: HBase Issue Type: Sub-task Reporter: stack Assignee: Andrew Purtell Currently, clients buried in REST server at least, are unable to recalibrate if the cluster is restarted on them. Fix it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4047) [Coprocessors] Generic external process host
[ https://issues.apache.org/jira/browse/HBASE-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170501#comment-13170501 ] Andrew Purtell commented on HBASE-4047: --- Start with testcases, the first a test that confirms a stuck child process via SIGSTOP doesn't take down the regionserver. Thinking there should be three selectable strategies: 1. Close and reopen the region, triggering force termination of the stuck child on close, and fork/initialization of a new child on open, along with reinit of all region related resources, other coprocessors, etc. 2. Unload/reload the malfunctioning coprocessor. Will require some work in the coprocessor framework to actually support unloading in a reasonable way. The JVM may make this complicated for integrated CPs, so perhaps just for those hosted in external processes. 3. Unload/terminate the malfunctioning coprocessor and continue operation. Consider changes in the CP framework for temporary blacklisting, will need that to avoid loading the suspect CP after a split. [Coprocessors] Generic external process host Key: HBASE-4047 URL: https://issues.apache.org/jira/browse/HBASE-4047 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: Andrew Purtell Assignee: Andrew Purtell Where HBase coprocessors deviate substantially from the design (as I understand it) of Google's BigTable coprocessors is we've reimagined it as a framework for internal extension. In contrast BigTable coprocessors run as separate processes colocated with tablet servers. The essential trade off is between performance, flexibility and possibility, and the ability to control and enforce resource usage. Since the initial design of HBase coprocessors some additional considerations are in play: - Developing computational frameworks sitting directly on top of HBase hosted in coprocessor(s); - Introduction of the map reduce next generation (mrng) resource management model, and the probability that limits will be enforced via cgroups at the OS level after this is generally available, e.g. when RHEL 6 deployments are common; - The possibility of deployment of HBase onto mrng-enabled Hadoop clusters via the mrng resource manager and a HBase-specific application controller. Therefore we should consider developing a coprocessor that is a generic host for another coprocessor, but one that forks a child process, loads the target coprocessor into the child, establishes a bidirectional pipe and uses an eventing model and umbilical protocol to provide for the coprocessor loaded into the child the same semantics as if it was loaded internally to the parent, and (eventually) use available resource management capabilities on the platform -- perhaps via the mrng resource controller or directly with cgroups -- to limit the child as desired by system administrators or the application designer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5047) [Coprocessors] Implement child failure strategies for external coprocessor host
[ https://issues.apache.org/jira/browse/HBASE-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170513#comment-13170513 ] Andrew Purtell commented on HBASE-5047: --- Is it sufficient to consider children that crashed as permanently stuck? [Coprocessors] Implement child failure strategies for external coprocessor host --- Key: HBASE-5047 URL: https://issues.apache.org/jira/browse/HBASE-5047 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell There should be three selectable strategies: 1. Close and reopen the region, triggering force termination of the stuck child on close, and fork/initialization of a new child on open, along with reinit of all region related resources, other coprocessors, etc. 2. Unload/reload the malfunctioning coprocessor. Will require some work in the coprocessor framework to actually support unloading in a reasonable way. The JVM may make this complicated for integrated CPs, so perhaps just for those hosted in external processes. 3. Unload/terminate the malfunctioning coprocessor and continue operation. Consider changes in the CP framework for temporary blacklisting, will need that to avoid loading the suspect CP after a split. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5047) [Coprocessors] Implement child failure strategies for external coprocessor host
[ https://issues.apache.org/jira/browse/HBASE-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170512#comment-13170512 ] Andrew Purtell commented on HBASE-5047: --- Also need a configurable option to declare laggy children as stuck, a hard response latency threshold and perhaps also a soft latency threshold with a tolerance factor, e.g. no more than 3 overages in a minute. [Coprocessors] Implement child failure strategies for external coprocessor host --- Key: HBASE-5047 URL: https://issues.apache.org/jira/browse/HBASE-5047 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell There should be three selectable strategies: 1. Close and reopen the region, triggering force termination of the stuck child on close, and fork/initialization of a new child on open, along with reinit of all region related resources, other coprocessors, etc. 2. Unload/reload the malfunctioning coprocessor. Will require some work in the coprocessor framework to actually support unloading in a reasonable way. The JVM may make this complicated for integrated CPs, so perhaps just for those hosted in external processes. 3. Unload/terminate the malfunctioning coprocessor and continue operation. Consider changes in the CP framework for temporary blacklisting, will need that to avoid loading the suspect CP after a split. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5047) [Coprocessors] Implement child failure strategies for external coprocessor host
[ https://issues.apache.org/jira/browse/HBASE-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170524#comment-13170524 ] Andrew Purtell commented on HBASE-5047: --- Regarding case #3 above, if a malfunctioning external coprocessor is terminated on one RS I think it makes most sense to propagate that failure to the whole cluster via a zookeeper based mechanism. Otherwise there are the obvious problems as a burden to the coprocessor developer. If choosing #3 as a strategy, the coprocessor developer has elected to take down the malfunctioning app but continue all other service. Allow resumption of service with a redeployment and an administrative action to reenable. [Coprocessors] Implement child failure strategies for external coprocessor host --- Key: HBASE-5047 URL: https://issues.apache.org/jira/browse/HBASE-5047 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Assignee: Andrew Purtell There should be three selectable strategies: 1. Close and reopen the region, triggering force termination of the stuck child on close, and fork/initialization of a new child on open, along with reinit of all region related resources, other coprocessors, etc. 2. Unload/reload the malfunctioning coprocessor. Will require some work in the coprocessor framework to actually support unloading in a reasonable way. The JVM may make this complicated for integrated CPs, so perhaps just for those hosted in external processes. 3. Unload/terminate the malfunctioning coprocessor and continue operation. Consider changes in the CP framework for temporary blacklisting, will need that to avoid loading the suspect CP after a split. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5040) Secure HBase builds fail
[
https://issues.apache.org/jira/browse/HBASE-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170531#comment-13170531
]
Andrew Purtell commented on HBASE-5040:
---
I'm +1 if then actually the build succeeds. Otherwise this issue can be an
umbrella for more work.
Secure HBase builds fail
Key: HBASE-5040
URL: https://issues.apache.org/jira/browse/HBASE-5040
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Fix For: 0.92.0
Attachments: 5040.txt
I saw the following in HBase-0.92-security build #39:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR]
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
[ERROR] - [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
{code}
The above was probably introduced by HBASE-5006
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5040) Secure HBase builds fail
[
https://issues.apache.org/jira/browse/HBASE-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170619#comment-13170619
]
Andrew Purtell commented on HBASE-5040:
---
We really should be building security with the same upstream Hadoop artifact as
otherwise. So while your change will get the security build past this breakage,
it's not the right fix IMO. We should get HADOOP-7070 into the next RC of 1.0,
if there is going to be one, or put up an updated patched version of Hadoop.
Secure HBase builds fail
Key: HBASE-5040
URL: https://issues.apache.org/jira/browse/HBASE-5040
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Fix For: 0.92.0
Attachments: 5040.txt
I saw the following in HBase-0.92-security build #39:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR]
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
[ERROR] - [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
{code}
The above was probably introduced by HBASE-5006
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5040) Secure HBase builds fail
[
https://issues.apache.org/jira/browse/HBASE-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170635#comment-13170635
]
Andrew Purtell commented on HBASE-5040:
---
Testing locally with an updated POM that changes the Hadoop artifact for
security builds from 0.20.205.1-7070-SNAPSHOT to 1.0.0rc2.
Secure HBase builds fail
Key: HBASE-5040
URL: https://issues.apache.org/jira/browse/HBASE-5040
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Fix For: 0.92.0
Attachments: 5040.txt
I saw the following in HBase-0.92-security build #39:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR]
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
[ERROR] - [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
https://builds.apache.org/job/HBase-0.92-security/ws/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java:[590,4]
method does not override or implement a method from a supertype
{code}
The above was probably introduced by HBASE-5006
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5026) Add coprocessor hook to HRegionServer.ScannerListener.leaseExpired()
[ https://issues.apache.org/jira/browse/HBASE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169596#comment-13169596 ] Andrew Purtell commented on HBASE-5026: --- +1 Any case where the CP framework misses a scanner close is a bug. Add coprocessor hook to HRegionServer.ScannerListener.leaseExpired() Key: HBASE-5026 URL: https://issues.apache.org/jira/browse/HBASE-5026 Project: HBase Issue Type: Bug Components: coprocessors Affects Versions: 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.92.0 Attachments: RegionObserverLeaseExpired.patch The RegionObserver's preScannerClose() and postScannerClose() methods should cover the scanner leaseExpired() situation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4990) Document secure HBase setup
[ https://issues.apache.org/jira/browse/HBASE-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167941#comment-13167941 ] Andrew Purtell commented on HBASE-4990: --- And I just realized I left out mention of EXEC privilege except for in the shell examples. This is included in the permissions enum but currently don't implement it. Will fix. Document secure HBase setup --- Key: HBASE-4990 URL: https://issues.apache.org/jira/browse/HBASE-4990 Project: HBase Issue Type: Sub-task Affects Versions: 0.92.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: 4990.txt, 4990.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4990) Document secure HBase setup
[ https://issues.apache.org/jira/browse/HBASE-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13166672#comment-13166672 ] Andrew Purtell commented on HBASE-4990: --- @stack: Thanks for looking at it. bq. Minor... something up w/ multiple additions to hbase-site.xml for server-side setup. You have two of them. I'll make it more clear what that is about in the next rev. Document secure HBase setup --- Key: HBASE-4990 URL: https://issues.apache.org/jira/browse/HBASE-4990 Project: HBase Issue Type: Sub-task Affects Versions: 0.92.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: 4990.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4938) Create a HRegion.getScanner public method that allows reading from a specified readPoint
[ https://issues.apache.org/jira/browse/HBASE-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162815#comment-13162815 ] Andrew Purtell commented on HBASE-4938: --- +1 Create a HRegion.getScanner public method that allows reading from a specified readPoint Key: HBASE-4938 URL: https://issues.apache.org/jira/browse/HBASE-4938 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Priority: Minor There is an existing api HRegion.getScanner(Scan) that allows scanning a table. My proposal is to extend it to HRegion.getScanner(Scan, long readPoint) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4946) HTable.coprocessorExec (and possibly coprocessorProxy) does not work with dynamically loaded coprocessors (from hdfs or local system), because the RPC system tries to d
[
https://issues.apache.org/jira/browse/HBASE-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162334#comment-13162334
]
Andrew Purtell commented on HBASE-4946:
---
Thanks for the patch Andrei, looks good. I gather from your comment above that
another version is forthcoming that contains a unit test, will hold off full
testing and commit until then.
HTable.coprocessorExec (and possibly coprocessorProxy) does not work with
dynamically loaded coprocessors (from hdfs or local system), because the RPC
system tries to deserialize an unknown class.
-
Key: HBASE-4946
URL: https://issues.apache.org/jira/browse/HBASE-4946
Project: HBase
Issue Type: Bug
Components: coprocessors
Affects Versions: 0.92.0
Reporter: Andrei Dragomir
Attachments: HBASE-4946.patch
Loading coprocessors jars from hdfs works fine. I load it from the shell,
after setting the attribute, and it gets loaded:
{noformat}
INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor
config now ...
INFO org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Class
com.MyCoprocessorClass needs to be loaded from a file -
hdfs://localhost:9000/coproc/rt- 0.0.1-SNAPSHOT.jar.
INFO org.apache.hadoop.hbase.coprocessor.CoprocessorHost: loadInstance:
com.MyCoprocessorClass
INFO org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost:
RegionEnvironment createEnvironment
DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
handler: region=t1,,1322572939753.6409aee1726d31f5e5671a59fe6e384f.
protocol=com.MyCoprocessorClassProtocol
INFO org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Load
coprocessor com.MyCoprocessorClass from HTD of t1 successfully.
{noformat}
The problem is that this coprocessors simply extends BaseEndpointCoprocessor,
with a dynamic method. When calling this method from the client with
HTable.coprocessorExec, I get errors on the HRegionServer, because the call
cannot be deserialized from writables.
The problem is that Exec tries to do an early resolve of the coprocessor
class. The coprocessor class is loaded, but it is in the context of the
HRegionServer / HRegion. So, the call fails:
{noformat}
2011-12-02 00:34:17,348 ERROR org.apache.hadoop.hbase.io.HbaseObjectWritable:
Error in readFields
java.io.IOException: Protocol class com.MyCoprocessorClassProtocol not found
at org.apache.hadoop.hbase.client.coprocessor.Exec.readFields(Exec.java:125)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:575)
at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:105)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1237)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1167)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:703)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:495)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.ClassNotFoundException: com.MyCoprocessorClassProtocol
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
at org.apache.hadoop.hbase.client.coprocessor.Exec.readFields(Exec.java:122)
... 10 more
{noformat}
Probably the correct way to fix this is to make Exec really smart, so that it
knows all the class definitions loaded in CoprocessorHost(s).
I created a small patch that simply doesn't resolve the class definition in
the Exec, instead passing it as string down to the HRegion layer. This layer
knows all the definitions, and simply loads it by name.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (HBASE-4944) Optionally verify bulk loaded HFiles
[ https://issues.apache.org/jira/browse/HBASE-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162528#comment-13162528 ] Andrew Purtell commented on HBASE-4944: --- Test failures are unrelated to the patch. All tests pass locally for me. @Ted, what do you think of v3? Optionally verify bulk loaded HFiles Key: HBASE-4944 URL: https://issues.apache.org/jira/browse/HBASE-4944 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Priority: Minor Attachments: HBASE-4944-v2.patch, HBASE-4944-v3.patch We rely on users to produce properly formatted HFiles for bulk import. Attached patch adds an optional code path, toggled by a configuration property, that verifies the HFile under consideration for import is properly sorted. The default maintains the current behavior, which does not scan the file for correctness. Patch is against trunk but can apply against all active branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4944) Optionally verify bulk loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162253#comment-13162253
]
Andrew Purtell commented on HBASE-4944:
---
From JIRA: Cannot attach file HBASE-4944.patch: Unknown server error (500).
The patch is pretty small, so here it is:
{code}
Index: src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
===
--- src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
(revision 1210044)
+++ src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
(working copy)
@@ -50,6 +50,7 @@
import org.apache.hadoop.hbase.io.hfile.Compression;
import org.apache.hadoop.hbase.io.hfile.HFile;
import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.io.hfile.InvalidHFileException;
import org.apache.hadoop.hbase.monitoring.MonitoredTask;
import org.apache.hadoop.hbase.regionserver.StoreScanner.ScanType;
import org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress;
@@ -123,6 +124,7 @@
private final String storeNameStr;
private CompactionProgress progress;
private final int compactionKVMax;
+ private final boolean verifyBulkLoads;
// not private for testing
/* package */ScanInfo scanInfo;
@@ -222,6 +224,9 @@
= conf.getLong(hbase.hstore.compaction.max.size, Long.MAX_VALUE);
this.compactionKVMax = conf.getInt(hbase.hstore.compaction.kv.max, 10);
+this.verifyBulkLoads = conf.getBoolean(hbase.hstore.bulkload.verify,
+true);
+
if (Store.closeCheckInterval == 0) {
Store.closeCheckInterval = conf.getInt(
hbase.hstore.close.check.interval, 10*1000*1000 /* 10 MB */);
@@ -355,8 +360,8 @@
}
/**
- * This throws a WrongRegionException if the bulkHFile does not fit in this
- * region.
+ * This throws a WrongRegionException if the HFile does not fit in this
+ * region, or an InvalidHFileException if the HFile is not valid.
*
*/
void assertBulkLoadHFileOk(Path srcPath) throws IOException {
@@ -386,6 +391,34 @@
Bulk load file + srcPath.toString() + does not fit inside
region
+ this.region);
}
+
+ if (verifyBulkLoads) {
+KeyValue pkv = null;
+HFileScanner scanner = reader.getScanner(false, false, false);
+scanner.seekTo();
+do {
+ KeyValue kv = scanner.getKeyValue();
+ if (pkv != null) {
+if (Bytes.compareTo(pkv.getBuffer(), pkv.getRowOffset(),
+pkv.getRowLength(), kv.getBuffer(), kv.getRowOffset(),
+kv.getRowLength()) 0) {
+ throw new InvalidHFileException(Previous row is greater then
+ + current row: path= + srcPath + previous=
+ + Bytes.toStringBinary(pkv.getKey()) + current=
+ + Bytes.toStringBinary(kv.getKey()));
+}
+if (Bytes.compareTo(pkv.getBuffer(), pkv.getFamilyOffset(),
+pkv.getFamilyLength(), kv.getBuffer(), kv.getFamilyOffset(),
+kv.getFamilyLength()) != 0) {
+ throw new InvalidHFileException(Previous key had different
+ + family compared to current key: path= + srcPath
+ + previous= + Bytes.toStringBinary(pkv.getKey())
+ + current= + Bytes.toStringBinary(kv.getKey()));
+}
+ }
+ pkv = kv;
+} while (scanner.next());
+ }
} finally {
if (reader != null) reader.close();
}
Index: src/main/java/org/apache/hadoop/hbase/io/hfile/InvalidHFileException.java
===
--- src/main/java/org/apache/hadoop/hbase/io/hfile/InvalidHFileException.java
(revision 0)
+++ src/main/java/org/apache/hadoop/hbase/io/hfile/InvalidHFileException.java
(revision 0)
@@ -0,0 +1,40 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import java.io.IOException;
+
+/**
+ * Thrown when an invalid HFile format is detected
+ */
+public class
[jira] [Commented] (HBASE-4942) HMaster is unable to start of HFile V1 is used
[
https://issues.apache.org/jira/browse/HBASE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13162296#comment-13162296
]
Andrew Purtell commented on HBASE-4942:
---
+1
HMaster is unable to start of HFile V1 is used
--
Key: HBASE-4942
URL: https://issues.apache.org/jira/browse/HBASE-4942
Project: HBase
Issue Type: Bug
Components: io
Affects Versions: 0.92.0
Reporter: Ted Yu
Assignee: honghua zhu
Fix For: 0.92.0, 0.94.0
Attachments: HBase_0.92.0_HBASE-4942, HBase_0.94.0_HBASE-4942
This was reported by HH Zhu (zhh200...@gmail.com)
If the following is specified in hbase-site.xml：
{code}
property
namehfile.format.version/name
value1/value
/property
{code}
Clear the hdfs directory hbase.rootdir so that MasterFileSystem.bootstrap()
is executed.
You would see:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV1.close(HFileReaderV1.java:358)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1083)
at
org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:570)
at org.apache.hadoop.hbase.regionserver.Store.close(Store.java:441)
at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:782)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:717)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:688)
at
org.apache.hadoop.hbase.master.MasterFileSystem.bootstrap(MasterFileSystem.java:390)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:356)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:128)
at
org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:113)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:435)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:314)
at java.lang.Thread.run(Thread.java:619)
{code}
The above exception would lead to:
{code}
java.lang.RuntimeException: HMaster Aborted
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:152)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1512)
{code}
In org.apache.hadoop.hbase.master.HMaster.HMaster(Configuration conf), we
have:
{code}
this.conf.setFloat(CacheConfig.HFILE_BLOCK_CACHE_SIZE_KEY, 0.0f);
{code}
When CacheConfig is instantiated, the following is called:
{code}
org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(Configuration
conf)
{code}
Since hfile.block.cache.size is 0.0, instantiateBlockCache() would return
null, resulting in blockCache field of CacheConfig to be null.
When master closes Root region,
org.apache.hadoop.hbase.io.hfile.HFileReaderV1.close(boolean evictOnClose)
would be called. cacheConf.getBlockCache() returns null, leading to master
abort.
The following should be called in HFileReaderV1.close(), similar to the code
in HFileReaderV2.close():
{code}
if (evictOnClose cacheConf.isBlockCacheEnabled())
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154276#comment-13154276 ] Andrew Purtell commented on HBASE-2418: --- @Mikhail: Thanks, that doesn't have a clear direct relation. If it were a test failure, I'd say otherwise. This patch modified the MiniZKCluster to take a Configuration in constructor and use it. This patch did not touch ZKConfig, which is HBase side code. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0, 0.94.0 Attachments: 2418.addendum, HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154280#comment-13154280 ] Andrew Purtell commented on HBASE-2418: --- @Ram I'm looking at the 0.92 pom right now and it includes the repository entry for ghelmling.testing. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0, 0.94.0 Attachments: 2418.addendum, HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154281#comment-13154281 ] Andrew Purtell commented on HBASE-2418: --- I opened HBASE-4835 for the CME. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0, 0.94.0 Attachments: 2418.addendum, HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4835) ConcurrentModificationException out of ZKConfig.makeZKProps
[
https://issues.apache.org/jira/browse/HBASE-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13154360#comment-13154360
]
Andrew Purtell commented on HBASE-4835:
---
Or synchronize access to the Configuration object in makeZKProps
ConcurrentModificationException out of ZKConfig.makeZKProps
---
Key: HBASE-4835
URL: https://issues.apache.org/jira/browse/HBASE-4835
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Attachments: HBASE-4835.patch
Mikhail reported this from a five-node, three-RS cluster test:
{code}
2011-11-21 01:30:15,188 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
machine_name,60020,1321867814890: Initialization of RS failed. Hence
aborting RS.
java.util.ConcurrentModificationException
at java.util.Hashtable$Enumerator.next(Hashtable.java:1031)
at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1042)
at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:75)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:144)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:124)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1262)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:568)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.init(HConnectionManager.java:559)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:183)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.init(CatalogTracker.java:177)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:575)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:534)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:642)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153921#comment-13153921 ] Andrew Purtell commented on HBASE-2418: --- Thanks Ted. I thought that went in with HBASE-3025. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0, 0.94.0 Attachments: 2418.addendum, HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153923#comment-13153923
]
Andrew Purtell commented on HBASE-2418:
---
And it looks like this part of the POM in trunk is not in the POM on 0.92:
{code}
pluginRepositories
pluginRepository
idghelmling.testing/id
nameGary Helmling test repo/name
urlhttp://people.apache.org/~garyh/mvn//url
snapshots
enabledtrue/enabled
/snapshots
releases
enabledtrue/enabled
/releases
/pluginRepository
/pluginRepositories
{code}
I don't know enough about Maven or how Gary set up the security profile to know
if it is needed or not. Gary?
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0, 0.94.0
Attachments: 2418.addendum, HBASE-2418-6.patch, HBASE-2418-6.patch
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically want to protect
their data (ZK znodes) from access by other services (tenants) on the
cluster. Say you are running HBase and Solr
and Neo4j, or multiple HBase instances, etc... having
authentication/authorization on the znodes is important for both
security and helping to ensure that services don't interact negatively (touch
each other's data).
Today HBase does not have support for authentication or authorization. This
should be added to the HBase clients
that are accessing the ZK cluster. In general it means calling addAuthInfo
once after a session is established:
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String,
byte[])
with a user specific credential, often times this is a shared secret or
certificate. You may be able to statically configure this
in some cases (config string or file to read from), however in my case in
particular you may need to access it programmatically,
which adds complexity as the end user may need to load code into HBase for
accessing the credential.
Secondly you need to specify a non world ACL when interacting with znodes
(create primarily):
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html
Feel free to ping the ZooKeeper team if you have questions. It might also be
good to discuss with some
potential end users - in particular regarding how the end user can specify
the credential.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153934#comment-13153934 ] Andrew Purtell commented on HBASE-2418: --- Hudson returned another build failure report. I committed the above to the 0.92 POM. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0, 0.94.0 Attachments: 2418.addendum, HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153538#comment-13153538
]
Andrew Purtell commented on HBASE-2418:
---
bq. Could the TestAdmin be because of this patch A?
I don't see it locally.
This change in the patch could be suspect but it's a shot in the dark:
{code}
--- src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
+++ src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
@@ -148,11 +156,14 @@ public class MiniZooKeeperCluster {
tickTimeToUse = TICK_TIME;
}
ZooKeeperServer server = new ZooKeeperServer(dir, dir, tickTimeToUse);
- NIOServerCnxn.Factory standaloneServerFactory;
+ NIOServerCnxnFactory standaloneServerFactory;
while (true) {
try {
- standaloneServerFactory = new NIOServerCnxn.Factory(
- new InetSocketAddress(tentativePort));
+ standaloneServerFactory = new NIOServerCnxnFactory();
+ standaloneServerFactory.configure(
+new InetSocketAddress(tentativePort),
+configuration.getInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS,
+ HConstants.DEFAULT_ZOOKEPER_MAX_CLIENT_CNXNS));
} catch (BindException e) {
LOG.debug(Failed binding ZK Server to client port: +
tentativePort);
{code}
I could change HConstants.DEFAULT_ZOOKEPER_MAX_CLIENT_CNXNS here to 1000 and
resubmit.
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0
Attachments: HBASE-2418-5.patch, HBASE-2418-5.patch,
HBASE-2418-5.patch
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically want to protect
their data (ZK znodes) from access by other services (tenants) on the
cluster. Say you are running HBase and Solr
and Neo4j, or multiple HBase instances, etc... having
authentication/authorization on the znodes is important for both
security and helping to ensure that services don't interact negatively (touch
each other's data).
Today HBase does not have support for authentication or authorization. This
should be added to the HBase clients
that are accessing the ZK cluster. In general it means calling addAuthInfo
once after a session is established:
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String,
byte[])
with a user specific credential, often times this is a shared secret or
certificate. You may be able to statically configure this
in some cases (config string or file to read from), however in my case in
particular you may need to access it programmatically,
which adds complexity as the end user may need to load code into HBase for
accessing the credential.
Secondly you need to specify a non world ACL when interacting with znodes
(create primarily):
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html
Feel free to ping the ZooKeeper team if you have questions. It might also be
good to discuss with some
potential end users - in particular regarding how the end user can specify
the credential.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149234#comment-13149234 ] Andrew Purtell commented on HBASE-3025: --- I'll sort out the disposition of this and the other two patches with Gary next week. With any luck we'll get them committed next week as well. Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025.2011-02-01.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4755) HBase based block placement in DFS
[ https://issues.apache.org/jira/browse/HBASE-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13147783#comment-13147783 ] Andrew Purtell commented on HBASE-4755: --- +1 We could also build upon this to locate read slaves (should that be separately pursued) on the secondary and tertiary node options. HBase based block placement in DFS -- Key: HBASE-4755 URL: https://issues.apache.org/jira/browse/HBASE-4755 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0 Reporter: Karthik Ranganathan Assignee: Christopher Gist The feature as is only useful for HBase clusters that care about data locality on regionservers, but this feature can also enable a lot of nice features down the road. The basic idea is as follows: instead of letting HDFS determine where to replicate data (r=3) by place blocks on various regions, it is better to let HBase do so by providing hints to HDFS through the DFS client. That way instead of replicating data at a blocks level, we can replicate data at a per-region level (each region owned by a promary, a secondary and a tertiary regionserver). This is better for 2 things: - Can make region failover faster on clusters which benefit from data affinity - On large clusters with random block placement policy, this helps reduce the probability of data loss The algo is as follows: - Each region in META will have 3 columns which are the preferred regionservers for that region (primary, secondary and tertiary) - Preferred assignment can be controlled by a config knob - Upon cluster start, HMaster will enter a mapping from each region to 3 regionservers (random hash, could use current locality, etc) - The load balancer would assign out regions preferring region assignments to primary over secondary over tertiary over any other node - Periodically (say weekly, configurable) the HMaster would run a locality checked and make sure the map it has for region to regionservers is optimal. Down the road, this can be enhanced to control region placement in the following cases: - Mixed hardware SKU where some regionservers can hold fewer regions - Load balancing across tables where we dont want multiple regions of a table to get assigned to the same regionservers - Multi-tenancy, where we can restrict the assignment of the regions of some table to a subset of regionservers, so an abusive app cannot take down the whole HBase cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
