[jira] [Updated] (HBASE-6049) Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject()
[ https://issues.apache.org/jira/browse/HBASE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated HBASE-6049: --- Attachment: HBASE-6049-v3.patch @stack, yes, there was a mistake. updated the patch. Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject() Key: HBASE-6049 URL: https://issues.apache.org/jira/browse/HBASE-6049 Project: HBase Issue Type: Bug Components: io Affects Versions: 0.94.0 Reporter: Maryann Xue Attachments: HBASE-6049-v2.patch, HBASE-6049-v3.patch, HBASE-6049.patch An error case could be in Coprocessor AggregationClient, the median() function handles an empty region and returns a List Object with the first element as a Null value. NPE occurs in the RPC response stage and the response never gets sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6132) ColumnCountGetFilter PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Kim updated HBASE-6132:
Description:
Thanks to Anoop and Ramkrishna, here's what we found with FilterList
If I use FilterList to include ColumnCountGetFilter among other filters, the
returning Result has no keyvalues.
This problem seems to occur when specified column count is less then actual
number of existing columns.
Also same problem arises with PageFilter
Following is the code of the problem:
{code}
Configuration conf = HBaseConfiguration.create();
HTable table = new HTable(conf, test);
Get get = new Get(Bytes.toBytes(test1));
FilterList filterList = new FilterList();
filterList.addFilter(new ColumnCountGetFilter(100));
get.setFilter(filterList);
Result r = table.get(get);
System.out.println(r.size()); // prints zero
{code}
was:
Thanks to Anoop and Ramkrishna, here's what we found with FilterList
If I use FilterList to include ColumnCountGetFilter among other filters, the
returning Result has no keyvalues.
This problem seems to occur when specified column count is less then actual
number of existing columns.
Also same problem arises with PageFilter
Following is the code of the problem:
Configuration conf = HBaseConfiguration.create();
HTable table = new HTable(conf, test);
Get get = new Get(Bytes.toBytes(test1));
FilterList filterList = new FilterList();
filterList.addFilter(new ColumnCountGetFilter(100));
get.setFilter(filterList);
Result r = table.get(get);
System.out.println(r.size()); // prints zero
ColumnCountGetFilter PageFilter not working with FilterList
-
Key: HBASE-6132
URL: https://issues.apache.org/jira/browse/HBASE-6132
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.92.0, 0.92.1, 0.94.0
Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
zookeeper 3.4.3
Reporter: Benjamin Kim
Thanks to Anoop and Ramkrishna, here's what we found with FilterList
If I use FilterList to include ColumnCountGetFilter among other filters, the
returning Result has no keyvalues.
This problem seems to occur when specified column count is less then actual
number of existing columns.
Also same problem arises with PageFilter
Following is the code of the problem:
{code}
Configuration conf = HBaseConfiguration.create();
HTable table = new HTable(conf, test);
Get get = new Get(Bytes.toBytes(test1));
FilterList filterList = new FilterList();
filterList.addFilter(new ColumnCountGetFilter(100));
get.setFilter(filterList);
Result r = table.get(get);
System.out.println(r.size()); // prints zero
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6096) AccessController v2
[
https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285418#comment-13285418
]
Laxman commented on HBASE-6096:
---
{quote}2) if you grant for 'A' you don't get RWC
so admin are not able to read but are able to perform actions
(create/delete/modify) on all tables{quote}
bq. IMO, it's preferable to conceptualize ADMIN permission as only an extra bit
that allows you to interact with the Master on table management concerns.
I agree. But I still need some details to prepare ACL matrix.
There are two levels of 'A' (ADMIN) now.
a) GLOBAL ADMIN (Super user - Configured using hbase.superuser)
b) TABLE ADMIN (via grant)
I'm not able to clearly differentiate between them.
In the current implementation, GLOBAL ADMIN is able to read/write to any table.
Which means, TABLE ADMIN will not be able to perform some operations on a table
which GLOBAL ADMIN is able to do. Now we may need to discuss which part of
these needs correction?
IMO, GLOBAL ADMIN (for all tables) semantics should be inline with TABLE ADMIN
(for one table).
AccessController v2
---
Key: HBASE-6096
URL: https://issues.apache.org/jira/browse/HBASE-6096
Project: HBase
Issue Type: Umbrella
Components: security
Affects Versions: 0.96.0, 0.94.1
Reporter: Andrew Purtell
Umbrella issue for iteration on the initial AccessController drop.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6096) AccessController v2
[ https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285420#comment-13285420 ] Laxman commented on HBASE-6096: --- bq. Read - checkAndPut/checkAndDelete @Matteo, I feel these two requires READ and WRITE together. No? AccessController v2 --- Key: HBASE-6096 URL: https://issues.apache.org/jira/browse/HBASE-6096 Project: HBase Issue Type: Umbrella Components: security Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Umbrella issue for iteration on the initial AccessController drop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6089) SSH and AM.joinCluster causes Concurrent Modification exception.
[
https://issues.apache.org/jira/browse/HBASE-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285421#comment-13285421
]
Anoop Sam John commented on HBASE-6089:
---
{quote}
So you suggest to change both the data structures this.servers and this.regions
to concurrentskiplistmap
{quote}
As per the current code written don't think we can change Ram.
We need a block of code to be sync which deals with both regions and servers
data structures. Also some other blocks where more than one operation on the
regions are being in sync block. Changing the data structure to
concurrentskiplistmap will not give this behaviour right.
SSH and AM.joinCluster causes Concurrent Modification exception.
Key: HBASE-6089
URL: https://issues.apache.org/jira/browse/HBASE-6089
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-6089_92.patch, HBASE-6089_94.patch,
HBASE-6089_trunk.patch
AM.regions map is parallely accessed in SSH and Master initialization leading
to ConcurrentModificationException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6088) Region splitting not happened for long time due to ZK exception while creating RS_ZK_SPLITTING node
[
https://issues.apache.org/jira/browse/HBASE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285423#comment-13285423
]
Zhihong Yu commented on HBASE-6088:
---
Addendum integrated to 0.94 branch.
Region splitting not happened for long time due to ZK exception while
creating RS_ZK_SPLITTING node
Key: HBASE-6088
URL: https://issues.apache.org/jira/browse/HBASE-6088
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: rajeshbabu
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-6088_92.patch, HBASE-6088_94.patch,
HBASE-6088_94_2.patch, HBASE-6088_94_3.patch, HBASE-6088_trunk.patch,
HBASE-6088_trunk_2.patch, HBASE-6088_trunk_3.patch, HBASE-6088_trunk_4.patch,
addendum_6088_94.patch
Region splitting not happened for long time due to ZK exception while
creating RS_ZK_SPLITTING node
{noformat}
2012-05-24 01:45:41,363 INFO org.apache.zookeeper.ClientCnxn: Client session
timed out, have not heard from server in 26668ms for sessionid
0x1377a75f41d0012, closing socket connection and attempting reconnect
2012-05-24 01:45:41,464 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase/unassigned/bd1079bf948c672e493432020dc0e144
{noformat}
{noformat}
2012-05-24 01:45:43,300 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
cleanupCurrentWriter waiting for transactions to get synced total 189377
synced till here 189365
2012-05-24 01:45:48,474 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup
of failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.; Failed
setting SPLITTING znode on
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
java.io.IOException: Failed setting SPLITTING znode on
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:242)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:450)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.zookeeper.KeeperException$BadVersionException:
KeeperErrorCode = BadVersion for
/hbase/unassigned/bd1079bf948c672e493432020dc0e144
at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1246)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:321)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:659)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:811)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:747)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.transitionNodeSplitting(SplitTransaction.java:919)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createNodeSplitting(SplitTransaction.java:869)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:239)
... 5 more
2012-05-24 01:45:48,476 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback of
failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
{noformat}
{noformat}
2012-05-24 01:47:28,141 ERROR
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/unassigned/bd1079bf948c672e493432020dc0e144 already exists and this is
not a retry
2012-05-24 01:47:28,142 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup
of failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.; Failed
create of ephemeral /hbase/unassigned/bd1079bf948c672e493432020dc0e144
java.io.IOException: Failed create of ephemeral
/hbase/unassigned/bd1079bf948c672e493432020dc0e144
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createNodeSplitting(SplitTransaction.java:865)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:239)
at
[jira] [Commented] (HBASE-6089) SSH and AM.joinCluster causes Concurrent Modification exception.
[
https://issues.apache.org/jira/browse/HBASE-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285425#comment-13285425
]
ramkrishna.s.vasudevan commented on HBASE-6089:
---
Yes. As per the javadoc for entrySet in ConcurrentSkipListMap
{code}
* pThe view's ttiterator/tt is a weakly consistent iterator
* that will never throw {@link ConcurrentModificationException},
* and guarantees to traverse elements as they existed upon
* construction of the iterator, and may (but is not guaranteed to)
* reflect any modifications subsequent to construction.
{code}
So as per the current JIRA when we iterate the this.region in joincluster we
need to get the actual regions there. If SSH tries to modify we may even try
to iterate those regions.
So making it to ConcurrentSkipListMap will not help us here and still we need
to go with the sync block.
So i think the current patch should be fine.
@Stack
What you feel? If it is ok for you , i can commit this?
SSH and AM.joinCluster causes Concurrent Modification exception.
Key: HBASE-6089
URL: https://issues.apache.org/jira/browse/HBASE-6089
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-6089_92.patch, HBASE-6089_94.patch,
HBASE-6089_trunk.patch
AM.regions map is parallely accessed in SSH and Master initialization leading
to ConcurrentModificationException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6131) Add attribution for code added by HBASE-5533 metrics
[ https://issues.apache.org/jira/browse/HBASE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285426#comment-13285426 ] Hadoop QA commented on HBASE-6131: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530161/6131.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2049//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2049//console This message is automatically generated. Add attribution for code added by HBASE-5533 metrics Key: HBASE-6131 URL: https://issues.apache.org/jira/browse/HBASE-6131 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Blocker Attachments: 6131.txt See the comment over in https://issues.apache.org/jira/browse/HBASE-5533?focusedCommentId=13283920page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13283920 The metrics histogram code was copied w/o attribution. Fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6088) Region splitting not happened for long time due to ZK exception while creating RS_ZK_SPLITTING node
[
https://issues.apache.org/jira/browse/HBASE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285427#comment-13285427
]
ramkrishna.s.vasudevan commented on HBASE-6088:
---
@Ted
Thanks a lot.
Region splitting not happened for long time due to ZK exception while
creating RS_ZK_SPLITTING node
Key: HBASE-6088
URL: https://issues.apache.org/jira/browse/HBASE-6088
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: rajeshbabu
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-6088_92.patch, HBASE-6088_94.patch,
HBASE-6088_94_2.patch, HBASE-6088_94_3.patch, HBASE-6088_trunk.patch,
HBASE-6088_trunk_2.patch, HBASE-6088_trunk_3.patch, HBASE-6088_trunk_4.patch,
addendum_6088_94.patch
Region splitting not happened for long time due to ZK exception while
creating RS_ZK_SPLITTING node
{noformat}
2012-05-24 01:45:41,363 INFO org.apache.zookeeper.ClientCnxn: Client session
timed out, have not heard from server in 26668ms for sessionid
0x1377a75f41d0012, closing socket connection and attempting reconnect
2012-05-24 01:45:41,464 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase/unassigned/bd1079bf948c672e493432020dc0e144
{noformat}
{noformat}
2012-05-24 01:45:43,300 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
cleanupCurrentWriter waiting for transactions to get synced total 189377
synced till here 189365
2012-05-24 01:45:48,474 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup
of failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.; Failed
setting SPLITTING znode on
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
java.io.IOException: Failed setting SPLITTING znode on
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:242)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:450)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.zookeeper.KeeperException$BadVersionException:
KeeperErrorCode = BadVersion for
/hbase/unassigned/bd1079bf948c672e493432020dc0e144
at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1246)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:321)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:659)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:811)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:747)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.transitionNodeSplitting(SplitTransaction.java:919)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createNodeSplitting(SplitTransaction.java:869)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:239)
... 5 more
2012-05-24 01:45:48,476 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback of
failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
{noformat}
{noformat}
2012-05-24 01:47:28,141 ERROR
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/unassigned/bd1079bf948c672e493432020dc0e144 already exists and this is
not a retry
2012-05-24 01:47:28,142 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup
of failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.; Failed
create of ephemeral /hbase/unassigned/bd1079bf948c672e493432020dc0e144
java.io.IOException: Failed create of ephemeral
/hbase/unassigned/bd1079bf948c672e493432020dc0e144
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createNodeSplitting(SplitTransaction.java:865)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:239)
at
[jira] [Created] (HBASE-6134) Speed up distribued split log
chunhui shen created HBASE-6134: --- Summary: Speed up distribued split log Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6049) Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject()
[ https://issues.apache.org/jira/browse/HBASE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285428#comment-13285428 ] Hadoop QA commented on HBASE-6049: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530165/HBASE-6049-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2050//console This message is automatically generated. Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject() Key: HBASE-6049 URL: https://issues.apache.org/jira/browse/HBASE-6049 Project: HBase Issue Type: Bug Components: io Affects Versions: 0.94.0 Reporter: Maryann Xue Attachments: HBASE-6049-v2.patch, HBASE-6049-v3.patch, HBASE-6049.patch An error case could be in Coprocessor AggregationClient, the median() function handles an empty region and returns a List Object with the first element as a Null value. NPE occurs in the RPC response stage and the response never gets sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6134) Speed up distribued split log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6134: Description: First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. Speed up distribued split log - Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6049) Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject()
[ https://issues.apache.org/jira/browse/HBASE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285434#comment-13285434 ] Zhihong Yu commented on HBASE-6049: --- @Maryann: Please rebase patch on latest trunk. Path for source would begin with hbase-server/ Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject() Key: HBASE-6049 URL: https://issues.apache.org/jira/browse/HBASE-6049 Project: HBase Issue Type: Bug Components: io Affects Versions: 0.94.0 Reporter: Maryann Xue Attachments: HBASE-6049-v2.patch, HBASE-6049-v3.patch, HBASE-6049.patch An error case could be in Coprocessor AggregationClient, the median() function handles an empty region and returns a List Object with the first element as a Null value. NPE occurs in the RPC response stage and the response never gets sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6083) Modify old filter tests to use Junit4/no longer use HBaseTestCase
[ https://issues.apache.org/jira/browse/HBASE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285436#comment-13285436 ] Hudson commented on HBASE-6083: --- Integrated in HBase-TRUNK #2952 (See [https://builds.apache.org/job/HBase-TRUNK/2952/]) HBASE-6083 Modify old filter tests to use Junit4/no longer use HBaseTestCase (Revision 1344111) Result = SUCCESS stack : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestBitComparator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestInclusiveStopFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestPageFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestRandomRowFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueExcludeFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java Modify old filter tests to use Junit4/no longer use HBaseTestCase - Key: HBASE-6083 URL: https://issues.apache.org/jira/browse/HBASE-6083 Project: HBase Issue Type: Improvement Reporter: Juhani Connolly Assignee: Juhani Connolly Priority: Minor Fix For: 0.96.0 Attachments: 6083v2.txt, 6083v2.txt, hbase-6083.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6114) CacheControl flags should be tunable per table schema per CF
[ https://issues.apache.org/jira/browse/HBASE-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285437#comment-13285437 ] Hudson commented on HBASE-6114: --- Integrated in HBase-TRUNK #2952 (See [https://builds.apache.org/job/HBase-TRUNK/2952/]) HBASE-6114. CacheControl flags should be tunable per table schema per CF (Revision 1344105) Result = SUCCESS apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java CacheControl flags should be tunable per table schema per CF Key: HBASE-6114 URL: https://issues.apache.org/jira/browse/HBASE-6114 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: 6114-0.92-v2.patch, 6114-0.92.patch, 6114-0.94-v2.patch, 6114-0.94.patch, 6114-trunk-v2.patch, 6114-trunk.patch, 6114.patch CacheControl flags should be tunable per table schema per CF, especially cacheDataOnWrite, also cacheIndexesOnWrite and cacheBloomsOnWrite. It looks like Store uses CacheConfig(Configuration conf, HColumnDescriptor family) to construct the CacheConfig, so it's a simple change there to override configuration properties with values of table schema attributes if present. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6114) CacheControl flags should be tunable per table schema per CF
[ https://issues.apache.org/jira/browse/HBASE-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285441#comment-13285441 ] Hudson commented on HBASE-6114: --- Integrated in HBase-0.94 #229 (See [https://builds.apache.org/job/HBase-0.94/229/]) HBASE-6114. CacheControl flags should be tunable per table schema per CF (Revision 1344106) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java CacheControl flags should be tunable per table schema per CF Key: HBASE-6114 URL: https://issues.apache.org/jira/browse/HBASE-6114 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: 6114-0.92-v2.patch, 6114-0.92.patch, 6114-0.94-v2.patch, 6114-0.94.patch, 6114-trunk-v2.patch, 6114-trunk.patch, 6114.patch CacheControl flags should be tunable per table schema per CF, especially cacheDataOnWrite, also cacheIndexesOnWrite and cacheBloomsOnWrite. It looks like Store uses CacheConfig(Configuration conf, HColumnDescriptor family) to construct the CacheConfig, so it's a simple change there to override configuration properties with values of table schema attributes if present. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Speed up distribued split log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285442#comment-13285442 ] Zhihong Yu commented on HBASE-6134: --- So your suggestion is to estimate log splitting duration based on number of HLog files ? If the estimate is low, we stay with master log splitting. Speed up distribued split log - Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6134) Speed up distribued split log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-6134: -- Fix Version/s: 0.96.0 Speed up distribued split log - Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6133) TestRestartCluster failing in 0.92
[ https://issues.apache.org/jira/browse/HBASE-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285453#comment-13285453 ] Hudson commented on HBASE-6133: --- Integrated in HBase-0.92 #430 (See [https://builds.apache.org/job/HBase-0.92/430/]) HBASE-6133 TestRestartCluster failing in 0.92 (Revision 1344096) Result = SUCCESS stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java TestRestartCluster failing in 0.92 -- Key: HBASE-6133 URL: https://issues.apache.org/jira/browse/HBASE-6133 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.92.2, 0.94.1 Attachments: waitonmaster.txt This test failed a few times just now on 0.92 branch. Seems pretty basic failure. The master is not up yet and its throwing PleaseHoldException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6088) Region splitting not happened for long time due to ZK exception while creating RS_ZK_SPLITTING node
[
https://issues.apache.org/jira/browse/HBASE-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285472#comment-13285472
]
Hudson commented on HBASE-6088:
---
Integrated in HBase-0.94 #230 (See
[https://builds.apache.org/job/HBase-0.94/230/])
HBASE-6088 Addendum fixes testSplitBeforeSettingSplittingInZK (Ram)
(Revision 1344113)
Result = FAILURE
tedyu :
Files :
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
Region splitting not happened for long time due to ZK exception while
creating RS_ZK_SPLITTING node
Key: HBASE-6088
URL: https://issues.apache.org/jira/browse/HBASE-6088
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Gopinathan A
Assignee: rajeshbabu
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-6088_92.patch, HBASE-6088_94.patch,
HBASE-6088_94_2.patch, HBASE-6088_94_3.patch, HBASE-6088_trunk.patch,
HBASE-6088_trunk_2.patch, HBASE-6088_trunk_3.patch, HBASE-6088_trunk_4.patch,
addendum_6088_94.patch
Region splitting not happened for long time due to ZK exception while
creating RS_ZK_SPLITTING node
{noformat}
2012-05-24 01:45:41,363 INFO org.apache.zookeeper.ClientCnxn: Client session
timed out, have not heard from server in 26668ms for sessionid
0x1377a75f41d0012, closing socket connection and attempting reconnect
2012-05-24 01:45:41,464 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase/unassigned/bd1079bf948c672e493432020dc0e144
{noformat}
{noformat}
2012-05-24 01:45:43,300 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
cleanupCurrentWriter waiting for transactions to get synced total 189377
synced till here 189365
2012-05-24 01:45:48,474 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup
of failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.; Failed
setting SPLITTING znode on
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
java.io.IOException: Failed setting SPLITTING znode on
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:242)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:450)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.zookeeper.KeeperException$BadVersionException:
KeeperErrorCode = BadVersion for
/hbase/unassigned/bd1079bf948c672e493432020dc0e144
at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1246)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:321)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:659)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:811)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:747)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.transitionNodeSplitting(SplitTransaction.java:919)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createNodeSplitting(SplitTransaction.java:869)
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:239)
... 5 more
2012-05-24 01:45:48,476 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback of
failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.
{noformat}
{noformat}
2012-05-24 01:47:28,141 ERROR
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/unassigned/bd1079bf948c672e493432020dc0e144 already exists and this is
not a retry
2012-05-24 01:47:28,142 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup
of failed split of
ufdr,011365398471659,1337823505339.bd1079bf948c672e493432020dc0e144.; Failed
create of ephemeral /hbase/unassigned/bd1079bf948c672e493432020dc0e144
java.io.IOException: Failed create of ephemeral
[jira] [Commented] (HBASE-5974) Scanner retry behavior with RPC timeout on next() seems incorrect
[ https://issues.apache.org/jira/browse/HBASE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285487#comment-13285487 ] Lars Hofhansl commented on HBASE-5974: -- W.r.t. to RegionScannerHolder. I see that makes sense. As long as the sequence numbering also works when the CP intercepts a scanner and provides its own implementation. I don't think there'd be a case where a CP would want to interfere with the sequence numbers. Re: the NoSuchMethodException check. I was wondering if there is anything with RPC version numbering to deal with this instead. Scanner retry behavior with RPC timeout on next() seems incorrect - Key: HBASE-5974 URL: https://issues.apache.org/jira/browse/HBASE-5974 Project: HBase Issue Type: Bug Components: client, regionserver Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0 Reporter: Todd Lipcon Assignee: Anoop Sam John Priority: Critical Fix For: 0.94.1 Attachments: HBASE-5974_0.94.patch, HBASE-5974_94-V2.patch I'm seeing the following behavior: - set RPC timeout to a short value - call next() for some batch of rows, big enough so the client times out before the result is returned - the HConnectionManager stuff will retry the next() call to the same server. At this point, one of two things can happen: 1) the previous next() call will still be processing, in which case you get a LeaseException, because it was removed from the map during the processing, or 2) the next() call will succeed but skip the prior batch of rows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5993) Add a no-read Append
[ https://issues.apache.org/jira/browse/HBASE-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285493#comment-13285493 ] Lars Hofhansl commented on HBASE-5993: -- Honestly, I still do not understand what Jacques is proposing. In order to append to something you'd have to read that something first. HBase has no in place updates (for a good reason). So one could: # Replace the KV if it is still in the memstore. # Store incremental changes (somewhere?) and combine upon read from HBase. Add a no-read Append Key: HBASE-5993 URL: https://issues.apache.org/jira/browse/HBASE-5993 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.0 Reporter: Jacques Priority: Critical HBASE-4102 added an atomic append. For high performance situations, it would be helpful to be able to do appends that don't actually require a read of the existing value. This would be useful in building a growing set of values. Our original use case was for implementing a form of search in HBase where a cell would contain a list of document ids associated with a particular keyword for search. However it seems like it would also be useful to provide substantial performance improvements for most Append scenarios. Within the client API, the simplest way to implement this would be to leverage the existing Append api. If the Append is marked as setReturnResults(false), use this code path. If result return is requested, use the existing Append implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6134) Speed up distribued split log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6134: Attachment: HBASE-6134.patch Speed up distribued split log - Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Speed up distribued split log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285496#comment-13285496 ] chunhui shen commented on HBASE-6134: - @ted In my opition, we could do the improvement for split-worker. In the patch , I change the split logic which executed by the split-worker same as local-master-split. It means, there are multi threads to creater writer and append hlog entry. Also, we found closing writer is quite slow, so make the closeStreams() executed by multi threads. With the patch, we test the time of distribued split log again, and reduce the time from 165s+ to 38s+ (34hlog files 4 regionervers) Speed up distribued split log - Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285498#comment-13285498 ] chunhui shen commented on HBASE-6134: - Review Board https://reviews.apache.org/r/5294/ Improvement for split-worker to speed up distributed-split-log -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6134: Summary: Improvement for split-worker to speed up distributed-split-log (was: Speed up distribued split log) Improvement for split-worker to speed up distributed-split-log -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285502#comment-13285502 ] ramkrishna.s.vasudevan commented on HBASE-6134: --- @Chunhui Nice work. bq.Also, we found closing writer is quite slow Any reason why it was slow? Improvement for split-worker to speed up distributed-split-log -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6116) Allow parallel HDFS writes for HLogs.
[ https://issues.apache.org/jira/browse/HBASE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285503#comment-13285503 ] Lars Hofhansl commented on HBASE-6116: -- Waiting for HDFS-1783 API to stabilize before I post a patch here. I don't currently have access to a real cluster, so if someone could do some testing in a real cluster with Hadoop-Trunk and HBase-Trunk, please let me know! Allow parallel HDFS writes for HLogs. - Key: HBASE-6116 URL: https://issues.apache.org/jira/browse/HBASE-6116 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl In HDFS-1783 I adapted Dhrubas changes to be used in Hadoop trunk. This issue will include the necessary reflection changes to optionally enable this for the WALs in HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5993) Add a no-read Append
[ https://issues.apache.org/jira/browse/HBASE-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285504#comment-13285504 ] Jieshan Bean commented on HBASE-5993: - One KeyValue will include 2 parts: The original value, and the appended value. So if we append new value, just add this value into the additional part(Maybe we can implement this by introducing a new KeyValue.Type). So we need to combine the values. (@Jacques: Please correct me if I misunderstood anything). I'm wondering whether it is really necessary. Add a no-read Append Key: HBASE-5993 URL: https://issues.apache.org/jira/browse/HBASE-5993 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.0 Reporter: Jacques Priority: Critical HBASE-4102 added an atomic append. For high performance situations, it would be helpful to be able to do appends that don't actually require a read of the existing value. This would be useful in building a growing set of values. Our original use case was for implementing a form of search in HBase where a cell would contain a list of document ids associated with a particular keyword for search. However it seems like it would also be useful to provide substantial performance improvements for most Append scenarios. Within the client API, the simplest way to implement this would be to leverage the existing Append api. If the Append is marked as setReturnResults(false), use this code path. If result return is requested, use the existing Append implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285511#comment-13285511 ] chunhui shen commented on HBASE-6134: - @ram Through the log, we found writer.close() took about 30ms~50ms. So, it will took about 20s if there are 400 regions in one hlog file. Improvement for split-worker to speed up distributed-split-log -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285513#comment-13285513 ] chunhui shen commented on HBASE-6134: - @ram Could you take a look at your environment, hom much time writer.close() took? Through the log: LOG.info(Closed path + wap.p + (wrote + wap.editsWritten + edits in + (wap.nanosSpent / 1000 / 1000) + ms)); Improvement for split-worker to speed up distributed-split-log -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285515#comment-13285515 ] ramkrishna.s.vasudevan commented on HBASE-6134: --- Ok. I will check that but may not be today. :) Improvement for split-worker to speed up distributed-split-log -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6114) CacheControl flags should be tunable per table schema per CF
[ https://issues.apache.org/jira/browse/HBASE-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285524#comment-13285524 ] Hudson commented on HBASE-6114: --- Integrated in HBase-0.92 #431 (See [https://builds.apache.org/job/HBase-0.92/431/]) HBASE-6114. CacheControl flags should be tunable per table schema per CF (Revision 1344107) Result = SUCCESS apurtell : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java CacheControl flags should be tunable per table schema per CF Key: HBASE-6114 URL: https://issues.apache.org/jira/browse/HBASE-6114 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: 6114-0.92-v2.patch, 6114-0.92.patch, 6114-0.94-v2.patch, 6114-0.94.patch, 6114-trunk-v2.patch, 6114-trunk.patch, 6114.patch CacheControl flags should be tunable per table schema per CF, especially cacheDataOnWrite, also cacheIndexesOnWrite and cacheBloomsOnWrite. It looks like Store uses CacheConfig(Configuration conf, HColumnDescriptor family) to construct the CacheConfig, so it's a simple change there to override configuration properties with values of table schema attributes if present. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5974) Scanner retry behavior with RPC timeout on next() seems incorrect
[ https://issues.apache.org/jira/browse/HBASE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285532#comment-13285532 ] Anoop Sam John commented on HBASE-5974: --- @Lars Or is it this kind of a compatibility is not needed? Only upgradation at client side and not at server. I am not sure what kind of compatibility we need to maintain across new minor versions (from 94.0 to 94.1) Also see the comments for the issue HBASE-6124. [Similar to our case] https://issues.apache.org/jira/browse/HBASE-6124?focusedCommentId=13285008page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13285008 https://issues.apache.org/jira/browse/HBASE-6124?focusedCommentId=13285016page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13285016 Scanner retry behavior with RPC timeout on next() seems incorrect - Key: HBASE-5974 URL: https://issues.apache.org/jira/browse/HBASE-5974 Project: HBase Issue Type: Bug Components: client, regionserver Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0 Reporter: Todd Lipcon Assignee: Anoop Sam John Priority: Critical Fix For: 0.94.1 Attachments: HBASE-5974_0.94.patch, HBASE-5974_94-V2.patch I'm seeing the following behavior: - set RPC timeout to a short value - call next() for some batch of rows, big enough so the client times out before the result is returned - the HConnectionManager stuff will retry the next() call to the same server. At this point, one of two things can happen: 1) the previous next() call will still be processing, in which case you get a LeaseException, because it was removed from the map during the processing, or 2) the next() call will succeed but skip the prior batch of rows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6089) SSH and AM.joinCluster causes Concurrent Modification exception.
[
https://issues.apache.org/jira/browse/HBASE-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285541#comment-13285541
]
ramkrishna.s.vasudevan commented on HBASE-6089:
---
Thinking again on this, there are some places where we wait for the
this.regions to be populated.
{code}
{code}
SSH and AM.joinCluster causes Concurrent Modification exception.
Key: HBASE-6089
URL: https://issues.apache.org/jira/browse/HBASE-6089
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-6089_92.patch, HBASE-6089_94.patch,
HBASE-6089_trunk.patch
AM.regions map is parallely accessed in SSH and Master initialization leading
to ConcurrentModificationException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6089) SSH and AM.joinCluster causes Concurrent Modification exception.
[
https://issues.apache.org/jira/browse/HBASE-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285541#comment-13285541
]
ramkrishna.s.vasudevan edited comment on HBASE-6089 at 5/30/12 10:09 AM:
-
I was not clear in the previous comment. Thinking again on this, there are
some places where we wait for the this.regions to be populated.
{code}
synchronized(regions) {
while(!regions.containsKey(regionInfo)) {
// We should receive a notification, but it's
// better to have a timeout to recheck the condition here:
// it lowers the impact of a race condition if any
regions.wait(100);
}
{code}
When we need to update the this.servers also along with this.regions then we
need to have some sync block. I agree with Anoop here.
was (Author: ram_krish):
Thinking again on this, there are some places where we wait for the
this.regions to be populated.
{code}
{code}
SSH and AM.joinCluster causes Concurrent Modification exception.
Key: HBASE-6089
URL: https://issues.apache.org/jira/browse/HBASE-6089
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-6089_92.patch, HBASE-6089_94.patch,
HBASE-6089_trunk.patch
AM.regions map is parallely accessed in SSH and Master initialization leading
to ConcurrentModificationException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5733) AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE.
[ https://issues.apache.org/jira/browse/HBASE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5733: -- Fix Version/s: 0.94.1 0.92.2 AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE. - Key: HBASE-5733 URL: https://issues.apache.org/jira/browse/HBASE-5733 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-5733.patch, HBASE-5733.patch, HBASE-5733.patch Found while going through the code... AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE as this is directly iterating the nodes from listChildrenAndWatchForNewChildren with-out checking for null. Here also we need to handle with null check like other places. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5733) AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE.
[ https://issues.apache.org/jira/browse/HBASE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285551#comment-13285551 ] ramkrishna.s.vasudevan commented on HBASE-5733: --- I think its better we commit it to 0.94.1 also before Lars could take the Rc. AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE. - Key: HBASE-5733 URL: https://issues.apache.org/jira/browse/HBASE-5733 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-5733.patch, HBASE-5733.patch, HBASE-5733.patch Found while going through the code... AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE as this is directly iterating the nodes from listChildrenAndWatchForNewChildren with-out checking for null. Here also we need to handle with null check like other places. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-5733) AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE.
[ https://issues.apache.org/jira/browse/HBASE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reopened HBASE-5733: --- Reopening so that once committed to other versions we can close it. AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE. - Key: HBASE-5733 URL: https://issues.apache.org/jira/browse/HBASE-5733 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-5733.patch, HBASE-5733.patch, HBASE-5733.patch Found while going through the code... AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE as this is directly iterating the nodes from listChildrenAndWatchForNewChildren with-out checking for null. Here also we need to handle with null check like other places. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6096) AccessController v2
[ https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285554#comment-13285554 ] Matteo Bertozzi commented on HBASE-6096: @Laxman if you look at TableAuthManager there're two SkipList USER_CACHE and TABLE_USER_CACHE USER_CACHE contains the global rights, TABLE_USER_CACHE contains the table one. hbase.superuser is read at startup TableAuthManager.initGlobal() and fills the USER_CACHE with specified users with RWCA rights (so is not just ADMIN is RWCA) All CA operation are checked against global rights (USER_CACHE) so specifying CA in table rights has no effect. grant 'user', 'RWCA' - fill USER_CACHE global rights grant 'user', 'RW', 'table' - FILL TABLE_USER_CACHE this means if you're global granted (CREATE/ADMIN) you can add/modify/remove tables with the restriction that, if you've CREATE you can just modify/delete your own table while ADMIN you can modify/delete all the available tables. if you're table granted you can do just read/write depends on what is your permission, but you don't have the ability to create/modify/delete even if you've CA (CA are not checked against TABLE_USER_CACHE just USER_CACHE) for checkAndPut()/checkAndDelete() there's a jira open HBASE-6062 and yes it needs to check both Read and Write. AccessController v2 --- Key: HBASE-6096 URL: https://issues.apache.org/jira/browse/HBASE-6096 Project: HBase Issue Type: Umbrella Components: security Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Umbrella issue for iteration on the initial AccessController drop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5722) NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW down.
[
https://issues.apache.org/jira/browse/HBASE-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5722:
--
Affects Version/s: 0.94.0
Fix Version/s: (was: 0.94.0)
0.94.1
0.96.0
NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW
down.
--
Key: HBASE-5722
URL: https://issues.apache.org/jira/browse/HBASE-5722
Project: HBase
Issue Type: Bug
Components: zookeeper
Affects Versions: 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Fix For: 0.96.0, 0.94.1
Attachments: HBASE-5722.patch
{code}
ListString nodes =
ZKUtil.listChildrenAndWatchForNewChildren(zkw, baseNode);
ListNodeAndData newNodes = new ArrayListNodeAndData();
for (String node: nodes) {
String nodePath = ZKUtil.joinZNode(baseNode, node);
byte [] data = ZKUtil.getDataAndWatch(zkw, nodePath);
newNodes.add(new NodeAndData(nodePath, data));
}
{code}
The above code can throw NPE when listChildrenAndWatchForNewChildren returns
null.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-5722) NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW down.
[
https://issues.apache.org/jira/browse/HBASE-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reopened HBASE-5722:
---
This got committed to 0.96 only and not to 0.94. Hence reopening so that once
committed we can close it.
NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW
down.
--
Key: HBASE-5722
URL: https://issues.apache.org/jira/browse/HBASE-5722
Project: HBase
Issue Type: Bug
Components: zookeeper
Affects Versions: 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Fix For: 0.96.0, 0.94.1
Attachments: HBASE-5722.patch
{code}
ListString nodes =
ZKUtil.listChildrenAndWatchForNewChildren(zkw, baseNode);
ListNodeAndData newNodes = new ArrayListNodeAndData();
for (String node: nodes) {
String nodePath = ZKUtil.joinZNode(baseNode, node);
byte [] data = ZKUtil.getDataAndWatch(zkw, nodePath);
newNodes.add(new NodeAndData(nodePath, data));
}
{code}
The above code can throw NPE when listChildrenAndWatchForNewChildren returns
null.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6062) preCheckAndPut/Delete() checks for READ when also a WRITE is performed
[ https://issues.apache.org/jira/browse/HBASE-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laxman updated HBASE-6062: -- Component/s: security preCheckAndPut/Delete() checks for READ when also a WRITE is performed -- Key: HBASE-6062 URL: https://issues.apache.org/jira/browse/HBASE-6062 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.92.1, 0.94.0, 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Labels: acl, security Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-6062-v0.patch preCheckAndPut() and preCheckAndDelete() checks for READ when they also want to WRITE... for me checking for WRITE permission is the right thing... what do you say about that? keep READ, replace with WRITE? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-6134: -- Status: Patch Available (was: Open) Improvement for split-worker to speed up distributed-split-log -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6083) Modify old filter tests to use Junit4/no longer use HBaseTestCase
[ https://issues.apache.org/jira/browse/HBASE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285604#comment-13285604 ] Hudson commented on HBASE-6083: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #32 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/32/]) HBASE-6083 Modify old filter tests to use Junit4/no longer use HBaseTestCase (Revision 1344111) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestBitComparator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestInclusiveStopFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestPageFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestRandomRowFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueExcludeFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java Modify old filter tests to use Junit4/no longer use HBaseTestCase - Key: HBASE-6083 URL: https://issues.apache.org/jira/browse/HBASE-6083 Project: HBase Issue Type: Improvement Reporter: Juhani Connolly Assignee: Juhani Connolly Priority: Minor Fix For: 0.96.0 Attachments: 6083v2.txt, 6083v2.txt, hbase-6083.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6043) Add Increment Coalescing in thrift.
[ https://issues.apache.org/jira/browse/HBASE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285603#comment-13285603 ] Hudson commented on HBASE-6043: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #32 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/32/]) HBASE-6043 Add Increment Coalescing in thrift. (Revision 1344086) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescerMBean.java Add Increment Coalescing in thrift. --- Key: HBASE-6043 URL: https://issues.apache.org/jira/browse/HBASE-6043 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-6043-0.patch, HBASE-6043-1.patch, HBASE-6043-10.patch, HBASE-6043-2.patch, HBASE-6043-3.patch, HBASE-6043-4.patch, HBASE-6043-5.patch, HBASE-6043-6.patch, HBASE-6043-7.patch, HBASE-6043-8.patch, HBASE-6043-9.patch, HBASE-6043-ADD.patch Since the thrift server uses the client api reducing the number of rpc's greatly speeds up increments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6133) TestRestartCluster failing in 0.92
[ https://issues.apache.org/jira/browse/HBASE-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285606#comment-13285606 ] Hudson commented on HBASE-6133: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #32 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/32/]) HBASE-6133 TestRestartCluster failing in 0.92 (Revision 1344098) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java TestRestartCluster failing in 0.92 -- Key: HBASE-6133 URL: https://issues.apache.org/jira/browse/HBASE-6133 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.92.2, 0.94.1 Attachments: waitonmaster.txt This test failed a few times just now on 0.92 branch. Seems pretty basic failure. The master is not up yet and its throwing PleaseHoldException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6114) CacheControl flags should be tunable per table schema per CF
[ https://issues.apache.org/jira/browse/HBASE-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285605#comment-13285605 ] Hudson commented on HBASE-6114: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #32 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/32/]) HBASE-6114. CacheControl flags should be tunable per table schema per CF (Revision 1344105) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java CacheControl flags should be tunable per table schema per CF Key: HBASE-6114 URL: https://issues.apache.org/jira/browse/HBASE-6114 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: 6114-0.92-v2.patch, 6114-0.92.patch, 6114-0.94-v2.patch, 6114-0.94.patch, 6114-trunk-v2.patch, 6114-trunk.patch, 6114.patch CacheControl flags should be tunable per table schema per CF, especially cacheDataOnWrite, also cacheIndexesOnWrite and cacheBloomsOnWrite. It looks like Store uses CacheConfig(Configuration conf, HColumnDescriptor family) to construct the CacheConfig, so it's a simple change there to override configuration properties with values of table schema attributes if present. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6096) AccessController v2
[ https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285611#comment-13285611 ] Laxman commented on HBASE-6096: --- @Matteo, Thanks a lot for detailed explanation. With your above explanation, I understand that current design/implementation ignores TABLE CREATE/ADMIN permissions. But here, I'm trying to get more on How it should be? than How it is currently? Found related discussion (TABLE ADMIN vs GLOBAL ADMIN) @ HBASE-5372. I'm sorry if I'm asking basic questions. bq. IMO, GLOBAL ADMIN (for all tables) semantics should be inline with TABLE ADMIN (for one table). Is this correct expectation or any different opinions? AccessController v2 --- Key: HBASE-6096 URL: https://issues.apache.org/jira/browse/HBASE-6096 Project: HBase Issue Type: Umbrella Components: security Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Umbrella issue for iteration on the initial AccessController drop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-6109: --- Attachment: 6109.v19.patch Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-6109: --- Status: Patch Available (was: Open) Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed-split-log
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285623#comment-13285623 ] Hadoop QA commented on HBASE-6134: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530177/HBASE-6134.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed org.apache.hadoop.hbase.regionserver.wal.TestWALReplay Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2051//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2051//console This message is automatically generated. Improvement for split-worker to speed up distributed-split-log -- Key: HBASE-6134 URL: https://issues.apache.org/jira/browse/HBASE-6134 Project: HBase Issue Type: Improvement Components: wal Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6134.patch First，we do the test between local-master-split and distributed split log Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work) local-master-split:60s+ distributed-split-log:165s+ In fact, in our production environment, distributed-split-log also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) We found split-worker split one log file took about 20s. I think we should do the improvement for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285640#comment-13285640 ] Hadoop QA commented on HBASE-6109: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530195/6109.v19.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2052//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2052//console This message is automatically generated. Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285671#comment-13285671 ] nkeywal commented on HBASE-6109: I think it's ok for a commit. From the code I read, we should have the same behavior as before on split. I will write some parallel tests later on, but I would expect the same behavior as today at least. It may take time as I may encounter some flakiness on this path ;-). I don't have a test class for NotifiableConcurrentSkipListMap, this class is small so I don't think it's an issue right now. I will push one with the other tests I will write. Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-6109: -- Fix Version/s: 0.96.0 Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6012) AssignmentManager#asyncSetOfflineInZooKeeper wouldn't force node offline
[
https://issues.apache.org/jira/browse/HBASE-6012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-6012:
--
Affects Version/s: 0.96.0
Fix Version/s: 0.96.0
AssignmentManager#asyncSetOfflineInZooKeeper wouldn't force node offline
Key: HBASE-6012
URL: https://issues.apache.org/jira/browse/HBASE-6012
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0
Attachments: HBASE-6012.patch
As the javadoc of method and the log message
{code}
/**
* Set region as OFFLINED up in zookeeper asynchronously.
*/
boolean asyncSetOfflineInZooKeeper(
...
master.abort(Unexpected ZK exception creating/setting node OFFLINE, e);
...
}
{code}
I think AssignmentManager#asyncSetOfflineInZooKeeper should also force node
offline, just like AssignmentManager#setOfflineInZooKeeper do. Otherwise, it
may cause bulk assign failed which called this method.
Error log on the master caused by the issue
2012-05-12 01:40:09,437 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=writetest,1YTQDPGLXBTICHOPQ6IL,1336590857771.674da422fc7cb9a7d42c74499ace1d93.
state=PENDING_CLOSE, ts=1336757876856
2012-05-12 01:40:09,437 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x23736bf74780082 Async create of unassigned node for
674da422fc7cb9a7d42c74499ace1d93 with OFFLINE state
2012-05-12 01:40:09,446 WARN
org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback:
rc != 0 for /hbase-func1/unassigned/674da422fc7cb9a7d42c74499ace1d93 --
retryable connectionloss -- FIX see
http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A2
2012-05-12 01:40:09,447 FATAL org.apache.hadoop.hbase.master.HMaster:
Connectionloss writing unassigned at
/hbase-func1/unassigned/674da422fc7cb9a7d42c74499ace1d93, rc=-110
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6012) AssignmentManager#asyncSetOfflineInZooKeeper wouldn't force node offline
[
https://issues.apache.org/jira/browse/HBASE-6012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285698#comment-13285698
]
ramkrishna.s.vasudevan commented on HBASE-6012:
---
I think this change we need to get into trunk after reviews for the SSH part to
work fine introduced in HBASE-5914.
AssignmentManager#asyncSetOfflineInZooKeeper wouldn't force node offline
Key: HBASE-6012
URL: https://issues.apache.org/jira/browse/HBASE-6012
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0
Attachments: HBASE-6012.patch
As the javadoc of method and the log message
{code}
/**
* Set region as OFFLINED up in zookeeper asynchronously.
*/
boolean asyncSetOfflineInZooKeeper(
...
master.abort(Unexpected ZK exception creating/setting node OFFLINE, e);
...
}
{code}
I think AssignmentManager#asyncSetOfflineInZooKeeper should also force node
offline, just like AssignmentManager#setOfflineInZooKeeper do. Otherwise, it
may cause bulk assign failed which called this method.
Error log on the master caused by the issue
2012-05-12 01:40:09,437 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=writetest,1YTQDPGLXBTICHOPQ6IL,1336590857771.674da422fc7cb9a7d42c74499ace1d93.
state=PENDING_CLOSE, ts=1336757876856
2012-05-12 01:40:09,437 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x23736bf74780082 Async create of unassigned node for
674da422fc7cb9a7d42c74499ace1d93 with OFFLINE state
2012-05-12 01:40:09,446 WARN
org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback:
rc != 0 for /hbase-func1/unassigned/674da422fc7cb9a7d42c74499ace1d93 --
retryable connectionloss -- FIX see
http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A2
2012-05-12 01:40:09,447 FATAL org.apache.hadoop.hbase.master.HMaster:
Connectionloss writing unassigned at
/hbase-func1/unassigned/674da422fc7cb9a7d42c74499ace1d93, rc=-110
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6131) Add attribution for code added by HBASE-5533 metrics
[ https://issues.apache.org/jira/browse/HBASE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6131: - Attachment: 6131_094.txt Here is what I applied to 0.94 branch. Add attribution for code added by HBASE-5533 metrics Key: HBASE-6131 URL: https://issues.apache.org/jira/browse/HBASE-6131 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Blocker Attachments: 6131.txt, 6131_094.txt See the comment over in https://issues.apache.org/jira/browse/HBASE-5533?focusedCommentId=13283920page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13283920 The metrics histogram code was copied w/o attribution. Fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6131) Add attribution for code added by HBASE-5533 metrics
[ https://issues.apache.org/jira/browse/HBASE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285735#comment-13285735 ] Hadoop QA commented on HBASE-6131: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530216/6131_094.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2053//console This message is automatically generated. Add attribution for code added by HBASE-5533 metrics Key: HBASE-6131 URL: https://issues.apache.org/jira/browse/HBASE-6131 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.94.1 Attachments: 6131.txt, 6131_094.txt See the comment over in https://issues.apache.org/jira/browse/HBASE-5533?focusedCommentId=13283920page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13283920 The metrics histogram code was copied w/o attribution. Fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6116) Allow parallel HDFS writes for HLogs.
[ https://issues.apache.org/jira/browse/HBASE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285760#comment-13285760 ] Andrew Purtell commented on HBASE-6116: --- When there is a patch I can try it out on a test cluster in EC2. Allow parallel HDFS writes for HLogs. - Key: HBASE-6116 URL: https://issues.apache.org/jira/browse/HBASE-6116 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl In HDFS-1783 I adapted Dhrubas changes to be used in Hadoop trunk. This issue will include the necessary reflection changes to optionally enable this for the WALs in HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5993) Add a no-read Append
[ https://issues.apache.org/jira/browse/HBASE-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285764#comment-13285764 ] Jacques commented on HBASE-5993: The reason this can make sense is data overhead. In a case where we are capturing a large number of small values, the KeyValue overhead is substantial. The original use case is one where I'm adding to a list of documents that contain a certain term (search index). Let's say that each document number is a four byte int. Right now there are two options: use the existing append which means one will become swamped with reads as the cell value grows over time (this would also wreak havoc on memstore flushes as the cell value become megabytes in size and we're just adding another four bytes once a day). On the flipside, using separate columns creates a substantial amount of overhead for each value added. This utility of this functionality also extends to situations where people are capturing a large sequence of small values in monitoring applications. (Sematext are basically trying to create this functionality already with their HBaseHUT work.) Yes, an additional KeyValue.Type is needed. When this type is read, the return functionality goes and get all the appended values (and the last full value) and then combines them on return. As compactions are done, the complete merged values are created. I'm swamped right now but am going to try to write up a short design doc in the next couple of weeks and get you guys to review my approach since this will have to touch a number of components. I also need to make sure to manage edge cases like what happens if you do a no-read append and no existing value exists (probably ok--even though read back performance will be poor). Add a no-read Append Key: HBASE-5993 URL: https://issues.apache.org/jira/browse/HBASE-5993 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.0 Reporter: Jacques Priority: Critical HBASE-4102 added an atomic append. For high performance situations, it would be helpful to be able to do appends that don't actually require a read of the existing value. This would be useful in building a growing set of values. Our original use case was for implementing a form of search in HBase where a cell would contain a list of document ids associated with a particular keyword for search. However it seems like it would also be useful to provide substantial performance improvements for most Append scenarios. Within the client API, the simplest way to implement this would be to leverage the existing Append api. If the Append is marked as setReturnResults(false), use this code path. If result return is requested, use the existing Append implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6096) AccessController v2
[ https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285766#comment-13285766 ] Andrew Purtell commented on HBASE-6096: --- bq. hbase.superuser is read at startup TableAuthManager.initGlobal() and fills the USER_CACHE with specified users with RWCA rights (so is not just ADMIN is RWCA) The superuser concept is a legacy from the initial implementation. Instead of having ADMIN rights mean something per user, there was/is an implicit grant of ADMIN rights to the superuser and that is it, for simplicity. So in our production the hbase user is used to manage the cluster by ops, and users have grants of only READ or WRITE as appropriate. bq. GLOBAL ADMIN (for all tables) semantics should be inline with TABLE ADMIN (for one table). +1 The superuser shortcut should be removed. Instead the AccessController could add a GLOBAL ADMIN grant on demand for the service principal of the master and regionservers when creating the ACL table. I don't think anyone is setting the superuser to anything other than the service principal. Also we could drop the owner concept (and table attribute) and instead have the AccessController add a TABLE ADMIN grant at table creation time, as discussed in HBASE-5372. AccessController v2 --- Key: HBASE-6096 URL: https://issues.apache.org/jira/browse/HBASE-6096 Project: HBase Issue Type: Umbrella Components: security Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Umbrella issue for iteration on the initial AccessController drop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6096) AccessController v2
[ https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285769#comment-13285769 ] Andrew Purtell commented on HBASE-6096: --- bq. checkAndPut/checkAndDelete These should need READ + WRITE. AccessController v2 --- Key: HBASE-6096 URL: https://issues.apache.org/jira/browse/HBASE-6096 Project: HBase Issue Type: Umbrella Components: security Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Umbrella issue for iteration on the initial AccessController drop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6096) AccessController v2
[ https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285771#comment-13285771 ] Andrew Purtell commented on HBASE-6096: --- And to clarify, the superuser can do anything, there is not only an implicit grant of ADMIN. AccessController v2 --- Key: HBASE-6096 URL: https://issues.apache.org/jira/browse/HBASE-6096 Project: HBase Issue Type: Umbrella Components: security Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Umbrella issue for iteration on the initial AccessController drop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6096) AccessController v2
[ https://issues.apache.org/jira/browse/HBASE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285771#comment-13285771 ] Andrew Purtell edited comment on HBASE-6096 at 5/30/12 4:02 PM: And to clarify, the superuser can do anything, there is not only an implicit grant of ADMIN. Now that people are working on functioning global permissions it's not necessary to have this shortcut. was (Author: apurtell): And to clarify, the superuser can do anything, there is not only an implicit grant of ADMIN. AccessController v2 --- Key: HBASE-6096 URL: https://issues.apache.org/jira/browse/HBASE-6096 Project: HBase Issue Type: Umbrella Components: security Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Umbrella issue for iteration on the initial AccessController drop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-6109: -- Attachment: 6049-v3.patch Patch v3 rebased on trunk. Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285780#comment-13285780 ] Andrew Purtell commented on HBASE-4720: --- +1 except for small debug logging nit that can be corrected on commit. Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.92.1, 0.96.0, 0.94.1 Attachments: 4720_trunk.patch, 4720_trunk_v2.patch, 4720_trunk_v3.patch, HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.trunk.v7.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-6109: -- Attachment: (was: 6049-v3.patch) Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285780#comment-13285780 ] Andrew Purtell edited comment on HBASE-4720 at 5/30/12 4:12 PM: +1 except for small debug logging nit that can be corrected on commit. See https://reviews.apache.org/r/5259/diff/3/?file=110632#file110632line294 was (Author: apurtell): +1 except for small debug logging nit that can be corrected on commit. Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.92.1, 0.96.0, 0.94.1 Attachments: 4720_trunk.patch, 4720_trunk_v2.patch, 4720_trunk_v3.patch, HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.trunk.v7.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6049) Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject()
[ https://issues.apache.org/jira/browse/HBASE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-6049: -- Attachment: 6049-v3.patch Patch v3 rebased on trunk. Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject() Key: HBASE-6049 URL: https://issues.apache.org/jira/browse/HBASE-6049 Project: HBase Issue Type: Bug Components: io Affects Versions: 0.94.0 Reporter: Maryann Xue Attachments: 6049-v3.patch, HBASE-6049-v2.patch, HBASE-6049-v3.patch, HBASE-6049.patch An error case could be in Coprocessor AggregationClient, the median() function handles an empty region and returns a List Object with the first element as a Null value. NPE occurs in the RPC response stage and the response never gets sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding
[ https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285793#comment-13285793 ] Jacques commented on HBASE-4676: Random thought on this. I was talking to Nicolas Spiegelberg and he was talking about how they are exploring changing encoding schemes when data becomes more permanent. Don't quote me on this but it sounded like they were considering using gzip compression on major compactions. I was wondering if something similar made sense here. Basically, use less compression the shorter the likely lifetime of files. Use the Trie compressions on only major compactions. This way the performance difference could be less of a serious issue since you're paying for it a fewer number of times. I glanced around and didn't see any Jiras about a two-tiered approach of data storage formats but it seems like that would be a prerequisite for a hybrid/tiered approach. Prefix Compression - Trie data block encoding - Key: HBASE-4676 URL: https://issues.apache.org/jira/browse/HBASE-4676 Project: HBase Issue Type: New Feature Components: io, performance, regionserver Affects Versions: 0.90.6 Reporter: Matt Corgan Assignee: Matt Corgan Attachments: HBASE-4676-0.94-v1.patch, PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by blockSize.png, hbase-prefix-trie-0.1.jar The HBase data block format has room for 2 significant improvements for applications that have high block cache hit ratios. First, there is no prefix compression, and the current KeyValue format is somewhat metadata heavy, so there can be tremendous memory bloat for many common data layouts, specifically those with long keys and short values. Second, there is no random access to KeyValues inside data blocks. This means that every time you double the datablock size, average seek time (or average cpu consumption) goes up by a factor of 2. The standard 64KB block size is ~10x slower for random seeks than a 4KB block size, but block sizes as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB or more may be more efficient from a disk access and block-cache perspective in many big-data applications, but doing so is infeasible from a random seek perspective. The PrefixTrie block encoding format attempts to solve both of these problems. Some features: * trie format for row key encoding completely eliminates duplicate row keys and encodes similar row keys into a standard trie structure which also saves a lot of space * the column family is currently stored once at the beginning of each block. this could easily be modified to allow multiple family names per block * all qualifiers in the block are stored in their own trie format which caters nicely to wide rows. duplicate qualifers between rows are eliminated. the size of this trie determines the width of the block's qualifier fixed-width-int * the minimum timestamp is stored at the beginning of the block, and deltas are calculated from that. the maximum delta determines the width of the block's timestamp fixed-width-int The block is structured with metadata at the beginning, then a section for the row trie, then the column trie, then the timestamp deltas, and then then all the values. Most work is done in the row trie, where every leaf node (corresponding to a row) contains a list of offsets/references corresponding to the cells in that row. Each cell is fixed-width to enable binary searching and is represented by [1 byte operationType, X bytes qualifier offset, X bytes timestamp delta offset]. If all operation types are the same for a block, there will be zero per-cell overhead. Same for timestamps. Same for qualifiers when i get a chance. So, the compression aspect is very strong, but makes a few small sacrifices on VarInt size to enable faster binary searches in trie fan-out nodes. A more compressed but slower version might build on this by also applying further (suffix, etc) compression on the trie nodes at the cost of slower write speed. Even further compression could be obtained by using all VInts instead of FInts with a sacrifice on random seek speed (though not huge). One current drawback is the current write speed. While programmed with good constructs like TreeMaps, ByteBuffers, binary searches, etc, it's not programmed with the same level of optimization as the read path. Work will need to be done to optimize the data structures used for encoding and could probably show a 10x increase. It will still be slower than delta encoding, but with a much higher decode speed. I have not yet created a thorough benchmark for write speed nor sequential read speed.
[jira] [Commented] (HBASE-5993) Add a no-read Append
[ https://issues.apache.org/jira/browse/HBASE-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285800#comment-13285800 ] Jacques commented on HBASE-5993: The implementation of HBASE-4218, HBASE-4676 and HBASE-6093 reduce the storage overhead of the multicolumn approach to solving this problem. Network bandwidth and processing overhead will still exist. Using encoding schemes to solve this problem is nice because the changes are constrained as opposed to cross-cutting. That being said, it seems a bit like boiling the ocean to make a cup of tea. Let me put a design doc together and then we can reevaluate. My intuition is that this type of functionality could open up a new set of use cases for HBase. Add a no-read Append Key: HBASE-5993 URL: https://issues.apache.org/jira/browse/HBASE-5993 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.0 Reporter: Jacques Priority: Critical HBASE-4102 added an atomic append. For high performance situations, it would be helpful to be able to do appends that don't actually require a read of the existing value. This would be useful in building a growing set of values. Our original use case was for implementing a form of search in HBase where a cell would contain a list of document ids associated with a particular keyword for search. However it seems like it would also be useful to provide substantial performance improvements for most Append scenarios. Within the client API, the simplest way to implement this would be to leverage the existing Append api. If the Append is marked as setReturnResults(false), use this code path. If result return is requested, use the existing Append implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-4720: --- Attachment: 4720_trunk_v3.1.patch Fix the logging issue Andrew found. Thanks Stack, Andrew, and Ted for reviewing. Thanks Mubarak for the patch. Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.92.1, 0.96.0, 0.94.1 Attachments: 4720_trunk.patch, 4720_trunk_v2.patch, 4720_trunk_v3.1.patch, 4720_trunk_v3.patch, HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.trunk.v7.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6091) Come up with strawman proposal for RC testing matrix
[ https://issues.apache.org/jira/browse/HBASE-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285804#comment-13285804 ] David S. Wang commented on HBASE-6091: -- I have some content ready; however is there some format I should post it in (e.g. Confluence wiki, etc.)? Come up with strawman proposal for RC testing matrix Key: HBASE-6091 URL: https://issues.apache.org/jira/browse/HBASE-6091 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.96.0 Reporter: David S. Wang Assignee: David S. Wang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6131) Add attribution for code added by HBASE-5533 metrics
[ https://issues.apache.org/jira/browse/HBASE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285808#comment-13285808 ] Hudson commented on HBASE-6131: --- Integrated in HBase-TRUNK #2953 (See [https://builds.apache.org/job/HBase-TRUNK/2953/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344299) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java * /hbase/trunk/pom.xml Add attribution for code added by HBASE-5533 metrics Key: HBASE-6131 URL: https://issues.apache.org/jira/browse/HBASE-6131 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.94.1 Attachments: 6131.txt, 6131_094.txt See the comment over in https://issues.apache.org/jira/browse/HBASE-5533?focusedCommentId=13283920page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13283920 The metrics histogram code was copied w/o attribution. Fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285807#comment-13285807 ] Hudson commented on HBASE-5533: --- Integrated in HBase-TRUNK #2953 (See [https://builds.apache.org/job/HBase-TRUNK/2953/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344299) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java * /hbase/trunk/pom.xml Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6109) Improve RIT performances during assignment on large clusters
[
https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285809#comment-13285809
]
Zhihong Yu commented on HBASE-6109:
---
Rename TestLocker class to TestKeyLocker ?
{code}
+// It has no reason to be a lock shares with the other operations.
{code}
'shares with' - 'shared with'
Indentation in AssignmentManager.addToRITandCallClose() was off. It would be
nice to correct the existing lines.
{code}
+// No lock concurrency: adding a share synchronized here would not prevent
to have two
+// entries as we don't check if the region is already there. This must be
ensured by the
+// method callers. todo nli: check
{code}
'share synchronized' - 'synchronized'. Remove the 'todo nli:' at the end.
{code}
-MapString, RegionPlan plans=new HashMapString, RegionPlan();
+MapString, RegionPlan plans=new HashMapString,
RegionPlan(regions.size());
{code}
Insert spaces around = sign.
{code}
+ * @return True if none of the regions in the set are in transition
{code}
'are in' - 'is in'
{code}
+ public NavigableMapK, V copyMap() {
+return delegatee.clone();
{code}
Why not call the method clone() ?
{code}
+ public void clear() {
+if (!delegatee.isEmpty()) {
+ synchronized (delegatee) {
{code}
Suppose delegatee is empty upon entry to the above method, what if an entry is
added after the isEmpty() check ?
Improve RIT performances during assignment on large clusters
Key: HBASE-6109
URL: https://issues.apache.org/jira/browse/HBASE-6109
Project: HBase
Issue Type: Improvement
Components: master
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Fix For: 0.96.0
Attachments: 6109.v19.patch, 6109.v7.patch
The main points in this patch are:
- lowering the number of copy of the RIT list
- lowering the number of synchronization
- synchronizing on a region rather than on everything
It also contains:
- some fixes around the RIT notification: the list was sometimes modified
without a corresponding 'notify'.
- some tests flakiness correction, actually unrelated to this patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285813#comment-13285813 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.94 #231 (See [https://builds.apache.org/job/HBase-0.94/231/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344305) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344304) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java stack : Files : * /hbase/branches/0.94/pom.xml * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6131) Add attribution for code added by HBASE-5533 metrics
[ https://issues.apache.org/jira/browse/HBASE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285814#comment-13285814 ] Hudson commented on HBASE-6131: --- Integrated in HBase-0.94 #231 (See [https://builds.apache.org/job/HBase-0.94/231/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344305) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344304) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java stack : Files : * /hbase/branches/0.94/pom.xml * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add attribution for code added by HBASE-5533 metrics Key: HBASE-6131 URL: https://issues.apache.org/jira/browse/HBASE-6131 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.94.1 Attachments: 6131.txt, 6131_094.txt See the comment over in https://issues.apache.org/jira/browse/HBASE-5533?focusedCommentId=13283920page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13283920 The metrics histogram code was copied w/o attribution. Fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6109) Improve RIT performances during assignment on large clusters
[
https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285821#comment-13285821
]
Zhihong Yu commented on HBASE-6109:
---
{code}
+ // A number of lock we want to easily support. It's not a maximum.
{code}
'A number' - 'The number'
{code}
+ // We need an atomic counter to manage the number of people using the lock
and free it when
+ // it's equals to zero.
{code}
'number of people' - 'number of users'
'it's equals to zero.' - 'it's equal to zero.'
{code}
+ static class RegionLockK extends ReentrantLock {
{code}
The outer class is generic. The inner class shouldn't mention Region.
Improve RIT performances during assignment on large clusters
Key: HBASE-6109
URL: https://issues.apache.org/jira/browse/HBASE-6109
Project: HBase
Issue Type: Improvement
Components: master
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Fix For: 0.96.0
Attachments: 6109.v19.patch, 6109.v7.patch
The main points in this patch are:
- lowering the number of copy of the RIT list
- lowering the number of synchronization
- synchronizing on a region rather than on everything
It also contains:
- some fixes around the RIT notification: the list was sometimes modified
without a corresponding 'notify'.
- some tests flakiness correction, actually unrelated to this patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6049) Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject()
[ https://issues.apache.org/jira/browse/HBASE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285822#comment-13285822 ] Hadoop QA commented on HBASE-6049: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530224/6049-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2054//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2054//console This message is automatically generated. Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject() Key: HBASE-6049 URL: https://issues.apache.org/jira/browse/HBASE-6049 Project: HBase Issue Type: Bug Components: io Affects Versions: 0.94.0 Reporter: Maryann Xue Attachments: 6049-v3.patch, HBASE-6049-v2.patch, HBASE-6049-v3.patch, HBASE-6049.patch An error case could be in Coprocessor AggregationClient, the median() function handles an empty region and returns a List Object with the first element as a Null value. NPE occurs in the RPC response stage and the response never gets sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6122) Backup master does not become Active master after ZK exception
[
https://issues.apache.org/jira/browse/HBASE-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-6122:
--
Fix Version/s: (was: 0.96.0)
Backup master does not become Active master after ZK exception
--
Key: HBASE-6122
URL: https://issues.apache.org/jira/browse/HBASE-6122
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.2, 0.94.1
Attachments: HBASE-6122_0.92.patch, HBASE-6122_0.94.patch
- Active master gets ZK expiry exception.
- Backup master becomes active.
- The previous active master retries and becomes the back up master.
Now when the new active master goes down and the current back up master comes
up, it goes down again with the zk expiry exception it got in the first step.
{code}
if (abortNow(msg, t)) {
if (t != null) LOG.fatal(msg, t);
else LOG.fatal(msg);
this.abort = true;
stop(Aborting);
}
{code}
In ActiveMasterManager.blockUntilBecomingActiveMaster we try to wait till the
back up master becomes active.
{code}
synchronized (this.clusterHasActiveMaster) {
while (this.clusterHasActiveMaster.get() !this.master.isStopped()) {
try {
this.clusterHasActiveMaster.wait();
} catch (InterruptedException e) {
// We expect to be interrupted when a master dies, will fall out if
so
LOG.debug(Interrupted waiting for master to die, e);
}
}
if (!clusterStatusTracker.isClusterUp()) {
this.master.stop(Cluster went down before this master became
active);
}
if (this.master.isStopped()) {
return cleanSetOfActiveMaster;
}
// Try to become active master again now that there is no active master
blockUntilBecomingActiveMaster(startupStatus,clusterStatusTracker);
}
return cleanSetOfActiveMaster;
{code}
When the back up master (it is in back up mode as he got ZK exception), once
again tries to come to active we don't get the return value that comes out
from
{code}
// Try to become active master again now that there is no active master
blockUntilBecomingActiveMaster(startupStatus,clusterStatusTracker);
{code}
We tend to return the 'cleanSetOfActiveMaster' which was previously false.
Now because of this instead of again becoming active the back up master goes
down in the abort() code. Thanks to Gopi,my colleague for reporting this
issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6049) Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject()
[ https://issues.apache.org/jira/browse/HBASE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-6049: -- Fix Version/s: 0.94.1 0.96.0 Hadoop Flags: Reviewed @Stack: Do you think patch v3 can be integrated ? Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject() Key: HBASE-6049 URL: https://issues.apache.org/jira/browse/HBASE-6049 Project: HBase Issue Type: Bug Components: io Affects Versions: 0.94.0 Reporter: Maryann Xue Fix For: 0.96.0, 0.94.1 Attachments: 6049-v3.patch, HBASE-6049-v2.patch, HBASE-6049-v3.patch, HBASE-6049.patch An error case could be in Coprocessor AggregationClient, the median() function handles an empty region and returns a List Object with the first element as a Null value. NPE occurs in the RPC response stage and the response never gets sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6122) Backup master does not become Active master after ZK exception
[
https://issues.apache.org/jira/browse/HBASE-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285823#comment-13285823
]
ramkrishna.s.vasudevan commented on HBASE-6122:
---
Committed to 0.92 and 0.94.
Thanks for the review Lars.
Backup master does not become Active master after ZK exception
--
Key: HBASE-6122
URL: https://issues.apache.org/jira/browse/HBASE-6122
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.2, 0.94.1
Attachments: HBASE-6122_0.92.patch, HBASE-6122_0.94.patch
- Active master gets ZK expiry exception.
- Backup master becomes active.
- The previous active master retries and becomes the back up master.
Now when the new active master goes down and the current back up master comes
up, it goes down again with the zk expiry exception it got in the first step.
{code}
if (abortNow(msg, t)) {
if (t != null) LOG.fatal(msg, t);
else LOG.fatal(msg);
this.abort = true;
stop(Aborting);
}
{code}
In ActiveMasterManager.blockUntilBecomingActiveMaster we try to wait till the
back up master becomes active.
{code}
synchronized (this.clusterHasActiveMaster) {
while (this.clusterHasActiveMaster.get() !this.master.isStopped()) {
try {
this.clusterHasActiveMaster.wait();
} catch (InterruptedException e) {
// We expect to be interrupted when a master dies, will fall out if
so
LOG.debug(Interrupted waiting for master to die, e);
}
}
if (!clusterStatusTracker.isClusterUp()) {
this.master.stop(Cluster went down before this master became
active);
}
if (this.master.isStopped()) {
return cleanSetOfActiveMaster;
}
// Try to become active master again now that there is no active master
blockUntilBecomingActiveMaster(startupStatus,clusterStatusTracker);
}
return cleanSetOfActiveMaster;
{code}
When the back up master (it is in back up mode as he got ZK exception), once
again tries to come to active we don't get the return value that comes out
from
{code}
// Try to become active master again now that there is no active master
blockUntilBecomingActiveMaster(startupStatus,clusterStatusTracker);
{code}
We tend to return the 'cleanSetOfActiveMaster' which was previously false.
Now because of this instead of again becoming active the back up master goes
down in the abort() code. Thanks to Gopi,my colleague for reporting this
issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6122) Backup master does not become Active master after ZK exception
[
https://issues.apache.org/jira/browse/HBASE-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan resolved HBASE-6122.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Backup master does not become Active master after ZK exception
--
Key: HBASE-6122
URL: https://issues.apache.org/jira/browse/HBASE-6122
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.2, 0.94.1
Attachments: HBASE-6122_0.92.patch, HBASE-6122_0.94.patch
- Active master gets ZK expiry exception.
- Backup master becomes active.
- The previous active master retries and becomes the back up master.
Now when the new active master goes down and the current back up master comes
up, it goes down again with the zk expiry exception it got in the first step.
{code}
if (abortNow(msg, t)) {
if (t != null) LOG.fatal(msg, t);
else LOG.fatal(msg);
this.abort = true;
stop(Aborting);
}
{code}
In ActiveMasterManager.blockUntilBecomingActiveMaster we try to wait till the
back up master becomes active.
{code}
synchronized (this.clusterHasActiveMaster) {
while (this.clusterHasActiveMaster.get() !this.master.isStopped()) {
try {
this.clusterHasActiveMaster.wait();
} catch (InterruptedException e) {
// We expect to be interrupted when a master dies, will fall out if
so
LOG.debug(Interrupted waiting for master to die, e);
}
}
if (!clusterStatusTracker.isClusterUp()) {
this.master.stop(Cluster went down before this master became
active);
}
if (this.master.isStopped()) {
return cleanSetOfActiveMaster;
}
// Try to become active master again now that there is no active master
blockUntilBecomingActiveMaster(startupStatus,clusterStatusTracker);
}
return cleanSetOfActiveMaster;
{code}
When the back up master (it is in back up mode as he got ZK exception), once
again tries to come to active we don't get the return value that comes out
from
{code}
// Try to become active master again now that there is no active master
blockUntilBecomingActiveMaster(startupStatus,clusterStatusTracker);
{code}
We tend to return the 'cleanSetOfActiveMaster' which was previously false.
Now because of this instead of again becoming active the back up master goes
down in the abort() code. Thanks to Gopi,my colleague for reporting this
issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5722) NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW down.
[
https://issues.apache.org/jira/browse/HBASE-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan resolved HBASE-5722.
---
Resolution: Fixed
NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW
down.
--
Key: HBASE-5722
URL: https://issues.apache.org/jira/browse/HBASE-5722
Project: HBase
Issue Type: Bug
Components: zookeeper
Affects Versions: 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Fix For: 0.96.0, 0.94.1
Attachments: HBASE-5722.patch
{code}
ListString nodes =
ZKUtil.listChildrenAndWatchForNewChildren(zkw, baseNode);
ListNodeAndData newNodes = new ArrayListNodeAndData();
for (String node: nodes) {
String nodePath = ZKUtil.joinZNode(baseNode, node);
byte [] data = ZKUtil.getDataAndWatch(zkw, nodePath);
newNodes.add(new NodeAndData(nodePath, data));
}
{code}
The above code can throw NPE when listChildrenAndWatchForNewChildren returns
null.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5722) NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW down.
[
https://issues.apache.org/jira/browse/HBASE-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285825#comment-13285825
]
ramkrishna.s.vasudevan commented on HBASE-5722:
---
The comment above mentioned that the commit was done to 0.94, but it wasn't. I
have committed it to 0.94. Just a mere copy paste. ensured that the test case
passes.
NPE in ZKUtil#getChildDataAndWatchForNewChildren when ZK not available or NW
down.
--
Key: HBASE-5722
URL: https://issues.apache.org/jira/browse/HBASE-5722
Project: HBase
Issue Type: Bug
Components: zookeeper
Affects Versions: 0.94.0, 0.96.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Fix For: 0.96.0, 0.94.1
Attachments: HBASE-5722.patch
{code}
ListString nodes =
ZKUtil.listChildrenAndWatchForNewChildren(zkw, baseNode);
ListNodeAndData newNodes = new ArrayListNodeAndData();
for (String node: nodes) {
String nodePath = ZKUtil.joinZNode(baseNode, node);
byte [] data = ZKUtil.getDataAndWatch(zkw, nodePath);
newNodes.add(new NodeAndData(nodePath, data));
}
{code}
The above code can throw NPE when listChildrenAndWatchForNewChildren returns
null.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5733) AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE.
[ https://issues.apache.org/jira/browse/HBASE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-5733. --- Resolution: Fixed Committed to 0.94 and 0.92. Hence resolving it. AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE. - Key: HBASE-5733 URL: https://issues.apache.org/jira/browse/HBASE-5733 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-5733.patch, HBASE-5733.patch, HBASE-5733.patch Found while going through the code... AssignmentManager#processDeadServersAndRegionsInTransition can fail with NPE as this is directly iterating the nodes from listChildrenAndWatchForNewChildren with-out checking for null. Here also we need to handle with null check like other places. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-6109: --- Status: Open (was: Patch Available) Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285828#comment-13285828 ] nkeywal commented on HBASE-6109: bq. Rename TestLocker class to TestKeyLocker ? Done. bq. 'shares with' - 'shared with' Done. bq. Indentation in AssignmentManager.addToRITandCallClose() was off. It would be nice to correct the existing lines. Done bq. 'share synchronized' - 'synchronized'. Remove the 'todo nli:' at the end. Done bq. Insert spaces around = sign. Done. bq. 'are in' - 'is in' Done bq. Why not call the method clone() ? We don't really want the NotifiableConcurrentSkipListMap to be cloneable: however, some functions want to work on a copy of the data structure, for reporting or test (with all the 'Map' semantic), hence the internal clone. bq. Suppose delegatee is empty upon entry to the above method, what if an entry is added after the isEmpty() check ? It will be equivalent to adding it just after the clear. bq. 'A number' - 'The number' Done. bq. 'number of people' - 'number of users' Done bq. 'it's equals to zero.' - 'it's equal to zero.' Done bq. The outer class is generic. The inner class shouldn't mention Region. Done Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-6109: --- Attachment: 6109.v21.patch Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v21.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding
[ https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285830#comment-13285830 ] Matt Corgan commented on HBASE-4676: Jacques - I like the idea , and I think the long-term goal is to auto-tune some of this stuff. For example, if flushes or compactions are backlogged, that maybe a good signal for certain tables to temporarily switch from more compact Trie/GZip encoding/compression to faster-encoding Prefix/Snappy. There are other factors at play though on a table-by-table basis. Some tables might be random-read heavy, in which case you would want to keep the Trie. Prefix Compression - Trie data block encoding - Key: HBASE-4676 URL: https://issues.apache.org/jira/browse/HBASE-4676 Project: HBase Issue Type: New Feature Components: io, performance, regionserver Affects Versions: 0.90.6 Reporter: Matt Corgan Assignee: Matt Corgan Attachments: HBASE-4676-0.94-v1.patch, PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by blockSize.png, hbase-prefix-trie-0.1.jar The HBase data block format has room for 2 significant improvements for applications that have high block cache hit ratios. First, there is no prefix compression, and the current KeyValue format is somewhat metadata heavy, so there can be tremendous memory bloat for many common data layouts, specifically those with long keys and short values. Second, there is no random access to KeyValues inside data blocks. This means that every time you double the datablock size, average seek time (or average cpu consumption) goes up by a factor of 2. The standard 64KB block size is ~10x slower for random seeks than a 4KB block size, but block sizes as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB or more may be more efficient from a disk access and block-cache perspective in many big-data applications, but doing so is infeasible from a random seek perspective. The PrefixTrie block encoding format attempts to solve both of these problems. Some features: * trie format for row key encoding completely eliminates duplicate row keys and encodes similar row keys into a standard trie structure which also saves a lot of space * the column family is currently stored once at the beginning of each block. this could easily be modified to allow multiple family names per block * all qualifiers in the block are stored in their own trie format which caters nicely to wide rows. duplicate qualifers between rows are eliminated. the size of this trie determines the width of the block's qualifier fixed-width-int * the minimum timestamp is stored at the beginning of the block, and deltas are calculated from that. the maximum delta determines the width of the block's timestamp fixed-width-int The block is structured with metadata at the beginning, then a section for the row trie, then the column trie, then the timestamp deltas, and then then all the values. Most work is done in the row trie, where every leaf node (corresponding to a row) contains a list of offsets/references corresponding to the cells in that row. Each cell is fixed-width to enable binary searching and is represented by [1 byte operationType, X bytes qualifier offset, X bytes timestamp delta offset]. If all operation types are the same for a block, there will be zero per-cell overhead. Same for timestamps. Same for qualifiers when i get a chance. So, the compression aspect is very strong, but makes a few small sacrifices on VarInt size to enable faster binary searches in trie fan-out nodes. A more compressed but slower version might build on this by also applying further (suffix, etc) compression on the trie nodes at the cost of slower write speed. Even further compression could be obtained by using all VInts instead of FInts with a sacrifice on random seek speed (though not huge). One current drawback is the current write speed. While programmed with good constructs like TreeMaps, ByteBuffers, binary searches, etc, it's not programmed with the same level of optimization as the read path. Work will need to be done to optimize the data structures used for encoding and could probably show a 10x increase. It will still be slower than delta encoding, but with a much higher decode speed. I have not yet created a thorough benchmark for write speed nor sequential read speed. Though the trie is reaching a point where it is internally very efficient (probably within half or a quarter of its max read speed) the way that hbase currently uses it is far from optimal. The KeyValueScanner and related classes that iterate through the trie will eventually need to be smarter
[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285831#comment-13285831 ] Hadoop QA commented on HBASE-4720: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530230/4720_trunk_v3.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager org.apache.hadoop.hbase.master.TestLogsCleaner org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2055//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2055//console This message is automatically generated. Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.92.1, 0.96.0, 0.94.1 Attachments: 4720_trunk.patch, 4720_trunk_v2.patch, 4720_trunk_v3.1.patch, 4720_trunk_v3.patch, HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.trunk.v7.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4720: - Status: Open (was: Patch Available) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.96.0, 0.94.1, 0.92.1 Attachments: 4720_trunk.patch, 4720_trunk_v2.patch, 4720_trunk_v3.1.patch, 4720_trunk_v3.patch, HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.trunk.v7.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285840#comment-13285840 ] Zhihong Yu commented on HBASE-6109: --- @N: Thanks for the quick turn-around. Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v21.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4720: - Attachment: 4720_trunk_v3.patch Retry just in case Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.92.1, 0.96.0, 0.94.1 Attachments: 4720_trunk.patch, 4720_trunk_v2.patch, 4720_trunk_v3.1.patch, 4720_trunk_v3.patch, 4720_trunk_v3.patch, HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.trunk.v7.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4720: - Status: Patch Available (was: Open) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord Assignee: Mubarak Seyed Fix For: 0.96.0, 0.94.1, 0.92.1 Attachments: 4720_trunk.patch, 4720_trunk_v2.patch, 4720_trunk_v3.1.patch, 4720_trunk_v3.patch, 4720_trunk_v3.patch, HBASE-4720.trunk.v1.patch, HBASE-4720.trunk.v2.patch, HBASE-4720.trunk.v3.patch, HBASE-4720.trunk.v4.patch, HBASE-4720.trunk.v5.patch, HBASE-4720.trunk.v6.patch, HBASE-4720.trunk.v7.patch, HBASE-4720.v1.patch, HBASE-4720.v3.patch I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6109: - Status: Patch Available (was: Open) Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v21.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6049) Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject()
[ https://issues.apache.org/jira/browse/HBASE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6049: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.94 and trunk. Thanks for the patch Maryann and thanks for rebasing Ted. Serializing List containing null elements will cause NullPointerException in HbaseObjectWritable.writeObject() Key: HBASE-6049 URL: https://issues.apache.org/jira/browse/HBASE-6049 Project: HBase Issue Type: Bug Components: io Affects Versions: 0.94.0 Reporter: Maryann Xue Fix For: 0.96.0, 0.94.1 Attachments: 6049-v3.patch, HBASE-6049-v2.patch, HBASE-6049-v3.patch, HBASE-6049.patch An error case could be in Coprocessor AggregationClient, the median() function handles an empty region and returns a List Object with the first element as a Null value. NPE occurs in the RPC response stage and the response never gets sent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5892) [hbck] Refactor parallel WorkItem* to Futures.
[ https://issues.apache.org/jira/browse/HBASE-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HBASE-5892: --- Attachment: hbase-5892-3.patch [hbck] Refactor parallel WorkItem* to Futures. -- Key: HBASE-5892 URL: https://issues.apache.org/jira/browse/HBASE-5892 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Assignee: Andrew Wang Labels: noob Attachments: hbase-5892-1.patch, hbase-5892-2-0.90.patch, hbase-5892-2.patch, hbase-5892-3.patch, hbase-5892.patch This would convert WorkItem* logic (with low level notifies, and rough exception handling) into a more canonical Futures pattern. Currently there are two instances of this pattern (for loading hdfs dirs, for contacting regionservers for assignments, and soon -- for loading hdfs .regioninfo files). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
