[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere
[ https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708257#comment-13708257 ] Enis Soztutar commented on HBASE-8927: -- bq. This was just to provide an example. Goal was to show that other applications are already using nano instead on milli, Cassandra's issue is about changing to nano for tracking latencies, vs. in this case, this is about using nano as a globally pseudo-consistent wall clock. Use nano time instead of mili time everywhere - Key: HBASE-8927 URL: https://issues.apache.org/jira/browse/HBASE-8927 Project: HBase Issue Type: Bug Reporter: stack Attachments: 8927.txt Less collisions and we are paying the price of a long anyways so might as well fill it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners
[
https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708288#comment-13708288
]
Devin Bayer commented on HBASE-7035:
If you read
[HTable|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html]
is says:
{quote}
This class is not thread safe for reads nor write.
...
In case of reads, some fields used by a Scan are shared among all threads. The
HTable implementation can either not contract to be safe in case of a Get
{quote}
That does not inspire confidence that scanners are or will remain threadsafe -
in fact it suggests the opposite.
thrift server closes HTable of open Scanners
Key: HBASE-7035
URL: https://issues.apache.org/jira/browse/HBASE-7035
Project: HBase
Issue Type: Sub-task
Components: Thrift
Affects Versions: 0.94.4
Reporter: Devin Bayer
Labels: thrift2
Attachments: old-hbase-thrift-v1.patch
Original Estimate: 4h
Remaining Estimate: 4h
ThriftHBaseServiceHandler.openScanner() does this:
1. table = pool.getTable()
2. scanner = table.getScanner()
3. table.close()
4. return scanner
While back porting the thrift server to 0.92.6, I found that table.close()
calls connection.close(). Further calls to scanner.next() raise a
ConnectionClosed exception. The unit tests do not catch this since they reuse
an open HConnection instance.
This might work on trunk, but depends on the implementations of HTablePool,
HTable and HConnectionManager. Even with the pool wrapper, if the pool is
full, table.close() may be called, which may invalidate the table. Also,
HTable is not thread-safe, but they are being reused since they go back in
the pool.
I suggest storing the table handle along with the scanner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners
[ https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708293#comment-13708293 ] Lars George commented on HBASE-7035: Hi Devin, I would not be surprised if this is outdated. I know that for writing the local client side write buffer is an issue since it is not thread safe. But for reading, this is news to me. Looking at the code, the ClientScanner has no reference to the HTable, and it does RPC requests to the RegionServers directly through the shared connection. I think we should raise a new JIRA to fix the JavaDoc for the HTable class and make sure it is current. thrift server closes HTable of open Scanners Key: HBASE-7035 URL: https://issues.apache.org/jira/browse/HBASE-7035 Project: HBase Issue Type: Sub-task Components: Thrift Affects Versions: 0.94.4 Reporter: Devin Bayer Labels: thrift2 Attachments: old-hbase-thrift-v1.patch Original Estimate: 4h Remaining Estimate: 4h ThriftHBaseServiceHandler.openScanner() does this: 1. table = pool.getTable() 2. scanner = table.getScanner() 3. table.close() 4. return scanner While back porting the thrift server to 0.92.6, I found that table.close() calls connection.close(). Further calls to scanner.next() raise a ConnectionClosed exception. The unit tests do not catch this since they reuse an open HConnection instance. This might work on trunk, but depends on the implementations of HTablePool, HTable and HConnectionManager. Even with the pool wrapper, if the pool is full, table.close() may be called, which may invalidate the table. Also, HTable is not thread-safe, but they are being reused since they go back in the pool. I suggest storing the table handle along with the scanner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners
[ https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708295#comment-13708295 ] Lars George commented on HBASE-7035: Looking at HBASE-4205, where this wording was added, it does not explain the reasons. For writing, this is clear, but not for reading. The [HTable\https://github.com/apache/hbase/blob/7f730d7742a523315db98d5adc879e67ed90350b/src/main/java/org/apache/hadoop/hbase/client/HTable.java] from back then (0.90.4) has changed a bit, and is now been given the connection reference directly, as opposed to be calling the method of the wrapping HTable instance. I'll raise this question up on dev@ to confirm the status quo. But, assuming the reading path is thread safe, we should be fine overall, no? Then using the pool is doing the job needed, keeping the connection open. In other words, for 0.94+ we do not need your patch? Not wanting to ruin the show here, please forgive me. :) thrift server closes HTable of open Scanners Key: HBASE-7035 URL: https://issues.apache.org/jira/browse/HBASE-7035 Project: HBase Issue Type: Sub-task Components: Thrift Affects Versions: 0.94.4 Reporter: Devin Bayer Labels: thrift2 Attachments: old-hbase-thrift-v1.patch Original Estimate: 4h Remaining Estimate: 4h ThriftHBaseServiceHandler.openScanner() does this: 1. table = pool.getTable() 2. scanner = table.getScanner() 3. table.close() 4. return scanner While back porting the thrift server to 0.92.6, I found that table.close() calls connection.close(). Further calls to scanner.next() raise a ConnectionClosed exception. The unit tests do not catch this since they reuse an open HConnection instance. This might work on trunk, but depends on the implementations of HTablePool, HTable and HConnectionManager. Even with the pool wrapper, if the pool is full, table.close() may be called, which may invalidate the table. Also, HTable is not thread-safe, but they are being reused since they go back in the pool. I suggest storing the table handle along with the scanner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7875) introduce a compaction switch in HBase Shell
[
https://issues.apache.org/jira/browse/HBASE-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Xie updated HBASE-7875:
-
Attachment: HBase-7875-v2.txt
rebase against trunk
introduce a compaction switch in HBase Shell
Key: HBASE-7875
URL: https://issues.apache.org/jira/browse/HBASE-7875
Project: HBase
Issue Type: New Feature
Components: shell
Affects Versions: 0.95.2
Reporter: Liang Xie
Assignee: Liang Xie
Priority: Trivial
Attachments: HBASE-7875.txt, HBASE-7875-v1.patch,
HBASE-7875-v1.patch, HBase-7875-v2.txt
A trivial patch to add an per-table compaction switch. it would be helpful
for OPS or manual testing in real cluster, etc.
Usage:
hbase(main):016:0 alter 'YCSBTest', {METHOD = 'table_att',
COMPACTION_ENABLE = 'FALSE'}
Updating all regions with the new schema...
64/64 regions updated.
Done.
0 row(s) in 1.1290 seconds
hbase(main):017:0 describe 'YCSBTest'
DESCRIPTION
ENABLED
{NAME = 'YCSBTest', COMPACTION_ENABLE = 'false', SPLIT_POLICY =
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPoli true
cy', FAMILIES = [{NAME = 'test', DATA_BLOCK_ENCODING = 'NONE',
BLOOMFILTER = 'NONE', REPLICATION_SCOPE = '0', COMPRESSION = 'S
NAPPY', VERSIONS = '1', TTL = '2147483647', MIN_VERSIONS = '0',
KEEP_DELETED_CELLS = 'false', BLOCKSIZE = '65536', ENCODE_ON_DI
SK = 'true', IN_MEMORY = 'false', BLOCKCACHE = 'true'}]}
1 row(s) in 0.0210 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7875) introduce a compaction switch in HBase Shell
[
https://issues.apache.org/jira/browse/HBASE-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708308#comment-13708308
]
Liang Xie commented on HBASE-7875:
--
[~sershe], [~yuzhih...@gmail.com], mind taking a look at it ? thanks
introduce a compaction switch in HBase Shell
Key: HBASE-7875
URL: https://issues.apache.org/jira/browse/HBASE-7875
Project: HBase
Issue Type: New Feature
Components: shell
Affects Versions: 0.95.2
Reporter: Liang Xie
Assignee: Liang Xie
Priority: Trivial
Attachments: HBASE-7875.txt, HBASE-7875-v1.patch,
HBASE-7875-v1.patch, HBase-7875-v2.txt
A trivial patch to add an per-table compaction switch. it would be helpful
for OPS or manual testing in real cluster, etc.
Usage:
hbase(main):016:0 alter 'YCSBTest', {METHOD = 'table_att',
COMPACTION_ENABLE = 'FALSE'}
Updating all regions with the new schema...
64/64 regions updated.
Done.
0 row(s) in 1.1290 seconds
hbase(main):017:0 describe 'YCSBTest'
DESCRIPTION
ENABLED
{NAME = 'YCSBTest', COMPACTION_ENABLE = 'false', SPLIT_POLICY =
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPoli true
cy', FAMILIES = [{NAME = 'test', DATA_BLOCK_ENCODING = 'NONE',
BLOOMFILTER = 'NONE', REPLICATION_SCOPE = '0', COMPRESSION = 'S
NAPPY', VERSIONS = '1', TTL = '2147483647', MIN_VERSIONS = '0',
KEEP_DELETED_CELLS = 'false', BLOCKSIZE = '65536', ENCODE_ON_DI
SK = 'true', IN_MEMORY = 'false', BLOCKCACHE = 'true'}]}
1 row(s) in 0.0210 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8928) Make ChaosMonkey LoadTest tools extensible, to allow addition of more actions and policies.
[ https://issues.apache.org/jira/browse/HBASE-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708314#comment-13708314 ] gautam commented on HBASE-8928: --- Here is the pull request. Again to avoid any confusion here: https://github.com/apache/hbase/pull/5 is over trunk branch https://github.com/apache/hbase/pull/4 is over 0.94 branch Elliott, I think you merged the 0.94 branch changes over trunk, hence it didnt work. Make ChaosMonkey LoadTest tools extensible, to allow addition of more actions and policies. - Key: HBASE-8928 URL: https://issues.apache.org/jira/browse/HBASE-8928 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.95.2 Reporter: gautam Assignee: Enis Soztutar Let me split this requirement into 2 parts: i) ChaosMonkey I was trying to add more tests around new actions and policies by leveraging the existing classes nested inside ChaosMonkey. But it turned out that some of the classes cannot be used outside, unless we make those visible to the world. Here is an example: I cannot extend ChaosMonkey.Action, as the init(ActionContext context) method has package-wide visibility. There are other places as well which makes it impossible for anyone to extend on top of this hierarchy. ii) LoadTestTool I wanted to extend this tool to define failure/pass criteria based on % of read/write failed, rather than comparing against absolute 0. For that this beautiful class should mark some of its properties usable by its child, by marking those protected. I wanted to get unblocked here first. Once this gets fixed, I think I can take up a JIRA item to refactor these tools, if required. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners
[ https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708328#comment-13708328 ] Devin Bayer commented on HBASE-7035: Hi Lars - what about this: If the pool is full, table.close() may be called by HTablePool. This calls connection.close(), but ResultScanner calls table.getConnection(). OK, maybe it will happen to work due to various coincidences, but it's not obvious it will work 100% of the time. For sure we must not change implementations of HTable, HConnection or HTablePool. We must also set the pool size greater than the thrift thread size. ClientScanner is an inner class and does retain a reference to the HTable instance. I think it's just asking for trouble. thrift server closes HTable of open Scanners Key: HBASE-7035 URL: https://issues.apache.org/jira/browse/HBASE-7035 Project: HBase Issue Type: Sub-task Components: Thrift Affects Versions: 0.94.4 Reporter: Devin Bayer Labels: thrift2 Attachments: old-hbase-thrift-v1.patch Original Estimate: 4h Remaining Estimate: 4h ThriftHBaseServiceHandler.openScanner() does this: 1. table = pool.getTable() 2. scanner = table.getScanner() 3. table.close() 4. return scanner While back porting the thrift server to 0.92.6, I found that table.close() calls connection.close(). Further calls to scanner.next() raise a ConnectionClosed exception. The unit tests do not catch this since they reuse an open HConnection instance. This might work on trunk, but depends on the implementations of HTablePool, HTable and HConnectionManager. Even with the pool wrapper, if the pool is full, table.close() may be called, which may invalidate the table. Also, HTable is not thread-safe, but they are being reused since they go back in the pool. I suggest storing the table handle along with the scanner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners
[ https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708333#comment-13708333 ] Lars George commented on HBASE-7035: bq. If the pool is full, table.close() may be called by HTablePool. This calls connection.close(), but ResultScanner calls table.getConnection(). No Devin, that was my point earlier, the connection.close() call use a reference counter. It does not close the connection while there are other users. And in case of Thrift, it has a pool still open and therefore the connection will *not* be closed under the hood. You do *not* have to set the pool size any different, or even larger at all. As long as the HTablePool is not closed, all is fine. thrift server closes HTable of open Scanners Key: HBASE-7035 URL: https://issues.apache.org/jira/browse/HBASE-7035 Project: HBase Issue Type: Sub-task Components: Thrift Affects Versions: 0.94.4 Reporter: Devin Bayer Labels: thrift2 Attachments: old-hbase-thrift-v1.patch Original Estimate: 4h Remaining Estimate: 4h ThriftHBaseServiceHandler.openScanner() does this: 1. table = pool.getTable() 2. scanner = table.getScanner() 3. table.close() 4. return scanner While back porting the thrift server to 0.92.6, I found that table.close() calls connection.close(). Further calls to scanner.next() raise a ConnectionClosed exception. The unit tests do not catch this since they reuse an open HConnection instance. This might work on trunk, but depends on the implementations of HTablePool, HTable and HConnectionManager. Even with the pool wrapper, if the pool is full, table.close() may be called, which may invalidate the table. Also, HTable is not thread-safe, but they are being reused since they go back in the pool. I suggest storing the table handle along with the scanner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7035) thrift server closes HTable of open Scanners
[ https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708333#comment-13708333 ] Lars George edited comment on HBASE-7035 at 7/15/13 9:10 AM: - bq. If the pool is full, table.close() may be called by HTablePool. This calls connection.close(), but ResultScanner calls table.getConnection(). No Devin, that was my point earlier, the connection.close() call use a reference counter. It does not close the connection while there are other users. And in case of Thrift, it has a pool still open and therefore the connection will *not* be closed under the hood. You do *not* have to set the pool size any different, or even larger at all. As long as the HTablePool is not closed, all is fine. was (Author: larsgeorge): bq. If the pool is full, table.close() may be called by HTablePool. This calls connection.close(), but ResultScanner calls table.getConnection(). No Devin, that was my point earlier, the connection.close() call use a reference counter. It does not close the connection while there are other users. And in case of Thrift, it has a pool still open and therefore the connection will *not* be closed under the hood. You do *not* have to set the pool size any different, or even larger at all. As long as the HTablePool is not closed, all is fine. thrift server closes HTable of open Scanners Key: HBASE-7035 URL: https://issues.apache.org/jira/browse/HBASE-7035 Project: HBase Issue Type: Sub-task Components: Thrift Affects Versions: 0.94.4 Reporter: Devin Bayer Labels: thrift2 Attachments: old-hbase-thrift-v1.patch Original Estimate: 4h Remaining Estimate: 4h ThriftHBaseServiceHandler.openScanner() does this: 1. table = pool.getTable() 2. scanner = table.getScanner() 3. table.close() 4. return scanner While back porting the thrift server to 0.92.6, I found that table.close() calls connection.close(). Further calls to scanner.next() raise a ConnectionClosed exception. The unit tests do not catch this since they reuse an open HConnection instance. This might work on trunk, but depends on the implementations of HTablePool, HTable and HConnectionManager. Even with the pool wrapper, if the pool is full, table.close() may be called, which may invalidate the table. Also, HTable is not thread-safe, but they are being reused since they go back in the pool. I suggest storing the table handle along with the scanner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7035) thrift server closes HTable of open Scanners
[ https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708295#comment-13708295 ] Lars George edited comment on HBASE-7035 at 7/15/13 9:10 AM: - Looking at HBASE-4205, where this wording was added, it does not explain the reasons. For writing, this is clear, but not for reading. The [HTable|https://github.com/apache/hbase/blob/7f730d7742a523315db98d5adc879e67ed90350b/src/main/java/org/apache/hadoop/hbase/client/HTable.java] from back then (0.90.4) has changed a bit, and is now been given the connection reference directly, as opposed to be calling the method of the wrapping HTable instance. I'll raise this question up on dev@ to confirm the status quo. But, assuming the reading path is thread safe, we should be fine overall, no? Then using the pool is doing the job needed, keeping the connection open. In other words, for 0.94+ we do not need your patch? Not wanting to ruin the show here, please forgive me. :) was (Author: larsgeorge): Looking at HBASE-4205, where this wording was added, it does not explain the reasons. For writing, this is clear, but not for reading. The [HTable\https://github.com/apache/hbase/blob/7f730d7742a523315db98d5adc879e67ed90350b/src/main/java/org/apache/hadoop/hbase/client/HTable.java] from back then (0.90.4) has changed a bit, and is now been given the connection reference directly, as opposed to be calling the method of the wrapping HTable instance. I'll raise this question up on dev@ to confirm the status quo. But, assuming the reading path is thread safe, we should be fine overall, no? Then using the pool is doing the job needed, keeping the connection open. In other words, for 0.94+ we do not need your patch? Not wanting to ruin the show here, please forgive me. :) thrift server closes HTable of open Scanners Key: HBASE-7035 URL: https://issues.apache.org/jira/browse/HBASE-7035 Project: HBase Issue Type: Sub-task Components: Thrift Affects Versions: 0.94.4 Reporter: Devin Bayer Labels: thrift2 Attachments: old-hbase-thrift-v1.patch Original Estimate: 4h Remaining Estimate: 4h ThriftHBaseServiceHandler.openScanner() does this: 1. table = pool.getTable() 2. scanner = table.getScanner() 3. table.close() 4. return scanner While back porting the thrift server to 0.92.6, I found that table.close() calls connection.close(). Further calls to scanner.next() raise a ConnectionClosed exception. The unit tests do not catch this since they reuse an open HConnection instance. This might work on trunk, but depends on the implementations of HTablePool, HTable and HConnectionManager. Even with the pool wrapper, if the pool is full, table.close() may be called, which may invalidate the table. Also, HTable is not thread-safe, but they are being reused since they go back in the pool. I suggest storing the table handle along with the scanner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8948) Bound table pool size in Thrift 2 server
Lars George created HBASE-8948:
--
Summary: Bound table pool size in Thrift 2 server
Key: HBASE-8948
URL: https://issues.apache.org/jira/browse/HBASE-8948
Project: HBase
Issue Type: Sub-task
Components: Thrift
Reporter: Lars George
Looking at the ThriftHBaseServiceHandler, it has this code:
{code}
ThriftHBaseServiceHandler(Configuration conf) {
htablePool = new HTablePool(conf, Integer.MAX_VALUE);
}
{code}
It seems like this is excessive. We should have a config to set this from
outside and default it to a sane upper limit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7035) thrift server closes HTable of open Scanners
[
https://issues.apache.org/jira/browse/HBASE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708337#comment-13708337
]
Lars George commented on HBASE-7035:
But this does point me to the actual pool size settings:
{code}
ThriftHBaseServiceHandler(Configuration conf) {
htablePool = new HTablePool(conf, Integer.MAX_VALUE);
}
{code}
which means it is pretty much unbound. That seems wrong too. I create
HBASE-8948 to track that separately.
thrift server closes HTable of open Scanners
Key: HBASE-7035
URL: https://issues.apache.org/jira/browse/HBASE-7035
Project: HBase
Issue Type: Sub-task
Components: Thrift
Affects Versions: 0.94.4
Reporter: Devin Bayer
Labels: thrift2
Attachments: old-hbase-thrift-v1.patch
Original Estimate: 4h
Remaining Estimate: 4h
ThriftHBaseServiceHandler.openScanner() does this:
1. table = pool.getTable()
2. scanner = table.getScanner()
3. table.close()
4. return scanner
While back porting the thrift server to 0.92.6, I found that table.close()
calls connection.close(). Further calls to scanner.next() raise a
ConnectionClosed exception. The unit tests do not catch this since they reuse
an open HConnection instance.
This might work on trunk, but depends on the implementations of HTablePool,
HTable and HConnectionManager. Even with the pool wrapper, if the pool is
full, table.close() may be called, which may invalidate the table. Also,
HTable is not thread-safe, but they are being reused since they go back in
the pool.
I suggest storing the table handle along with the scanner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8943) Split Thrift2's ThriftServer into separate classes for easier testing and modularization
[ https://issues.apache.org/jira/browse/HBASE-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708342#comment-13708342 ] Lars George commented on HBASE-8943: Hi Stack, I am not sure they can share much, they quickly get into the issue of linking to the specific Thrift modules. Also, Thrift 1 is going to go away eventually, so why start fiddling with a factory class of sorts? I am still amazed how we ripped out REST and replaced it with Stargate (aka REST 2) and no one asked these questions? Split Thrift2's ThriftServer into separate classes for easier testing and modularization Key: HBASE-8943 URL: https://issues.apache.org/jira/browse/HBASE-8943 Project: HBase Issue Type: Sub-task Components: Thrift Reporter: Lars George Assignee: Lars George Labels: thrift2 Currently the ThriftServer class in Thrift 2 sets up and starts the actual server. Better follow a similar pattern to Thrift 1 where there is some factory setting up the server, and a separate start section. That way it is easier to test if the setup of the server is picking up everything it needs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8928) Make ChaosMonkey LoadTest tools extensible, to allow addition of more actions and policies.
[ https://issues.apache.org/jira/browse/HBASE-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gautam updated HBASE-8928: -- Attachment: HBASE-8928-trunk.patch HBASE-8928-0.94.patch GIT Patch files - HBASE-8928-0.94.patch HBASE-8928-trunk.patch to apply to trunk and the 0.94 branches respectively. Make ChaosMonkey LoadTest tools extensible, to allow addition of more actions and policies. - Key: HBASE-8928 URL: https://issues.apache.org/jira/browse/HBASE-8928 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.95.2 Reporter: gautam Assignee: Enis Soztutar Attachments: HBASE-8928-0.94.patch, HBASE-8928-trunk.patch Let me split this requirement into 2 parts: i) ChaosMonkey I was trying to add more tests around new actions and policies by leveraging the existing classes nested inside ChaosMonkey. But it turned out that some of the classes cannot be used outside, unless we make those visible to the world. Here is an example: I cannot extend ChaosMonkey.Action, as the init(ActionContext context) method has package-wide visibility. There are other places as well which makes it impossible for anyone to extend on top of this hierarchy. ii) LoadTestTool I wanted to extend this tool to define failure/pass criteria based on % of read/write failed, rather than comparing against absolute 0. For that this beautiful class should mark some of its properties usable by its child, by marking those protected. I wanted to get unblocked here first. Once this gets fixed, I think I can take up a JIRA item to refactor these tools, if required. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7875) introduce a compaction switch in HBase Shell
[
https://issues.apache.org/jira/browse/HBASE-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708344#comment-13708344
]
Hadoop QA commented on HBASE-7875:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592292/HBase-7875-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor
org.apache.hadoop.hbase.regionserver.TestCompaction
org.apache.hadoop.hbase.client.TestShell
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6337//console
This message is automatically generated.
introduce a compaction switch in HBase Shell
Key: HBASE-7875
URL: https://issues.apache.org/jira/browse/HBASE-7875
Project: HBase
Issue Type: New Feature
Components: shell
Affects Versions: 0.95.2
Reporter: Liang Xie
Assignee: Liang Xie
Priority: Trivial
Attachments: HBASE-7875.txt, HBASE-7875-v1.patch,
HBASE-7875-v1.patch, HBase-7875-v2.txt
A trivial patch to add an per-table compaction switch. it would be helpful
for OPS or manual testing in real cluster, etc.
Usage:
hbase(main):016:0 alter 'YCSBTest', {METHOD = 'table_att',
COMPACTION_ENABLE = 'FALSE'}
Updating all regions with the new schema...
64/64 regions updated.
Done.
0 row(s) in 1.1290 seconds
hbase(main):017:0 describe 'YCSBTest'
DESCRIPTION
ENABLED
{NAME = 'YCSBTest', COMPACTION_ENABLE = 'false', SPLIT_POLICY =
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPoli true
cy', FAMILIES = [{NAME = 'test', DATA_BLOCK_ENCODING = 'NONE',
BLOOMFILTER = 'NONE', REPLICATION_SCOPE = '0', COMPRESSION = 'S
NAPPY', VERSIONS = '1', TTL = '2147483647', MIN_VERSIONS = '0',
KEEP_DELETED_CELLS = 'false', BLOCKSIZE = '65536', ENCODE_ON_DI
[jira] [Commented] (HBASE-8943) Split Thrift2's ThriftServer into separate classes for easier testing and modularization
[ https://issues.apache.org/jira/browse/HBASE-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708347#comment-13708347 ] Lars George commented on HBASE-8943: Just to clarify my above comment on REST - it seems that if we would have had someone in the past to drive Thrift, we would not be in this dual interface predicament. REST had an owner and got in timely and swiftly. Thrift should have been the same. No one to blame, just saying. Split Thrift2's ThriftServer into separate classes for easier testing and modularization Key: HBASE-8943 URL: https://issues.apache.org/jira/browse/HBASE-8943 Project: HBase Issue Type: Sub-task Components: Thrift Reporter: Lars George Assignee: Lars George Labels: thrift2 Currently the ThriftServer class in Thrift 2 sets up and starts the actual server. Better follow a similar pattern to Thrift 1 where there is some factory setting up the server, and a separate start section. That way it is easier to test if the setup of the server is picking up everything it needs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8937) createEphemeralNodeAndWatch don't set watcher if the node is created successfully
[
https://issues.apache.org/jira/browse/HBASE-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-8937:
---
Labels: master wacter (was: )
createEphemeralNodeAndWatch don't set watcher if the node is created
successfully
-
Key: HBASE-8937
URL: https://issues.apache.org/jira/browse/HBASE-8937
Project: HBase
Issue Type: Bug
Components: master
Reporter: Liu Shaohui
Priority: Minor
Labels: master, wacter
CreateEphemeralNodeAndWatch in zkUtil don't set watcher if the node is
created successfully. This is not consistent with the comment and may causes
the ActiveMasterManager cannot get events that master node is deleted or
changed.
{code}
public static boolean createEphemeralNodeAndWatch(ZooKeeperWatcher zkw,
String znode, byte [] data)
throws KeeperException {
try {
zkw.getRecoverableZooKeeper().create(znode, data, createACL(zkw, znode),
CreateMode.EPHEMERAL);
} catch (KeeperException.NodeExistsException nee) {
if(!watchAndCheckExists(zkw, znode)) {
// It did exist but now it doesn't, try again
return createEphemeralNodeAndWatch(zkw, znode, data);
}
return false;
} catch (InterruptedException e) {
LOG.info(Interrupted, e);
Thread.currentThread().interrupt();
}
return true;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8937) createEphemeralNodeAndWatch don't set watcher if the node is created successfully
[
https://issues.apache.org/jira/browse/HBASE-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-8937:
---
Component/s: master
createEphemeralNodeAndWatch don't set watcher if the node is created
successfully
-
Key: HBASE-8937
URL: https://issues.apache.org/jira/browse/HBASE-8937
Project: HBase
Issue Type: Bug
Components: master
Reporter: Liu Shaohui
Priority: Minor
CreateEphemeralNodeAndWatch in zkUtil don't set watcher if the node is
created successfully. This is not consistent with the comment and may causes
the ActiveMasterManager cannot get events that master node is deleted or
changed.
{code}
public static boolean createEphemeralNodeAndWatch(ZooKeeperWatcher zkw,
String znode, byte [] data)
throws KeeperException {
try {
zkw.getRecoverableZooKeeper().create(znode, data, createACL(zkw, znode),
CreateMode.EPHEMERAL);
} catch (KeeperException.NodeExistsException nee) {
if(!watchAndCheckExists(zkw, znode)) {
// It did exist but now it doesn't, try again
return createEphemeralNodeAndWatch(zkw, znode, data);
}
return false;
} catch (InterruptedException e) {
LOG.info(Interrupted, e);
Thread.currentThread().interrupt();
}
return true;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8348) Polish the migration to 0.96
[ https://issues.apache.org/jira/browse/HBASE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-8348: -- Attachment: HBASE-8348_trunk.patch Here is the patch for trunk which includes migration tool to 1) Migrate zookeeper data if no hbase process is alive ■ Find .META region server from -ROOT- and create meta-region-server ■ Migrate table states to PB(solve HBASE-7766 and helps to find exact state of a table otherwise even disabled tables also be will enabled during master startup.) ■ clean the znodes which are not used in 0.95.0 or not needed during clean cluster startup(znodes like rs,root-region-server,unassigned,splitlog,hbase-id,shutdown are removed). 2) remove -ROOT- table from filesystem 3) Identify HFiles of version 1(using HFileV1Detector tool). and also removed TestMetaMigrationRemovingHTD#testMetaUpdatedFlagInROOT [~saint@gmail.com] bq. Also ..test that 0.95 hlogs and hfiles are readable (from Matteo) Able to read hfiles and user table hlogs of older hbase version properly but not able to read META table hlogs because we are writing edits of it into a separate hlog suffixed by -meta. So not able to find it from hlogs od older version. I think It would be better to flush -ROOT- and .META. tables atleast before stopping the cluster. Please suggest some solution for this scenario. Please review the patch and provide any suggestions/comments. Polish the migration to 0.96 Key: HBASE-8348 URL: https://issues.apache.org/jira/browse/HBASE-8348 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.95.2 Attachments: HBASE-8348_trunk.patch Currently, migration works but there's still a couple of rough edges: - HBASE-8045 finished the .META. migration but didn't remove ROOT, so it's still on the filesystem. - Data in ZK needs to be removed manually. Either we fix up the data in ZK or we delete it ourselves. - TestMetaMigrationRemovingHTD has a testMetaUpdatedFlagInROOT method, but ROOT is gone now. Elliott was also mentioning that we could have hbase migrate do the HFileV1 checks, clear ZK, remove ROOT, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8348) Polish the migration to 0.96
[ https://issues.apache.org/jira/browse/HBASE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-8348: -- Status: Patch Available (was: Open) Polish the migration to 0.96 Key: HBASE-8348 URL: https://issues.apache.org/jira/browse/HBASE-8348 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.95.2 Attachments: HBASE-8348_trunk.patch Currently, migration works but there's still a couple of rough edges: - HBASE-8045 finished the .META. migration but didn't remove ROOT, so it's still on the filesystem. - Data in ZK needs to be removed manually. Either we fix up the data in ZK or we delete it ourselves. - TestMetaMigrationRemovingHTD has a testMetaUpdatedFlagInROOT method, but ROOT is gone now. Elliott was also mentioning that we could have hbase migrate do the HFileV1 checks, clear ZK, remove ROOT, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-8348) Polish the migration to 0.96
[ https://issues.apache.org/jira/browse/HBASE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-8348: - Assignee: rajeshbabu Polish the migration to 0.96 Key: HBASE-8348 URL: https://issues.apache.org/jira/browse/HBASE-8348 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: Jean-Daniel Cryans Assignee: rajeshbabu Priority: Critical Fix For: 0.95.2 Attachments: HBASE-8348_trunk.patch Currently, migration works but there's still a couple of rough edges: - HBASE-8045 finished the .META. migration but didn't remove ROOT, so it's still on the filesystem. - Data in ZK needs to be removed manually. Either we fix up the data in ZK or we delete it ourselves. - TestMetaMigrationRemovingHTD has a testMetaUpdatedFlagInROOT method, but ROOT is gone now. Elliott was also mentioning that we could have hbase migrate do the HFileV1 checks, clear ZK, remove ROOT, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8348) Polish the migration to 0.96
[
https://issues.apache.org/jira/browse/HBASE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708399#comment-13708399
]
Hadoop QA commented on HBASE-8348:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592303/HBASE-8348_trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6338//console
This message is automatically generated.
Polish the migration to 0.96
Key: HBASE-8348
URL: https://issues.apache.org/jira/browse/HBASE-8348
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0
Reporter: Jean-Daniel Cryans
Assignee: rajeshbabu
Priority: Critical
Fix For: 0.95.2
Attachments: HBASE-8348_trunk.patch
Currently, migration works but there's still a couple of rough edges:
- HBASE-8045 finished the .META. migration but didn't remove ROOT, so it's
still on the filesystem.
- Data in ZK needs to be removed manually. Either we fix up the data in ZK
or we delete it ourselves.
- TestMetaMigrationRemovingHTD has a testMetaUpdatedFlagInROOT method, but
ROOT is gone now.
Elliott was also mentioning that we could have hbase migrate do the HFileV1
checks, clear ZK, remove ROOT, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8348) Polish the migration to 0.96
[ https://issues.apache.org/jira/browse/HBASE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-8348: -- Attachment: HBASE-8348_trunk_v2.patch setting default fs in HFileV1Detector tool. Polish the migration to 0.96 Key: HBASE-8348 URL: https://issues.apache.org/jira/browse/HBASE-8348 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: Jean-Daniel Cryans Assignee: rajeshbabu Priority: Critical Fix For: 0.95.2 Attachments: HBASE-8348_trunk.patch, HBASE-8348_trunk_v2.patch Currently, migration works but there's still a couple of rough edges: - HBASE-8045 finished the .META. migration but didn't remove ROOT, so it's still on the filesystem. - Data in ZK needs to be removed manually. Either we fix up the data in ZK or we delete it ourselves. - TestMetaMigrationRemovingHTD has a testMetaUpdatedFlagInROOT method, but ROOT is gone now. Elliott was also mentioning that we could have hbase migrate do the HFileV1 checks, clear ZK, remove ROOT, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8432) a table with unbalanced regions will balance indefinitely with the 'org.apache.hadoop.hbase.master.DefaultLoadBalancer'
[ https://issues.apache.org/jira/browse/HBASE-8432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708406#comment-13708406 ] Jean-Marc Spaggiari commented on HBASE-8432: Hi Wang, Have you tried this change with the 3 different scenarios: 1) More regions than RS; 2) More regions than RS/2 but less than RS; 3) Less regions than RS/2. ? a table with unbalanced regions will balance indefinitely with the 'org.apache.hadoop.hbase.master.DefaultLoadBalancer' --- Key: HBASE-8432 URL: https://issues.apache.org/jira/browse/HBASE-8432 Project: HBase Issue Type: Bug Components: Balancer Affects Versions: 0.94.5 Environment: Linux 2.6.32-el5.x86_64 Reporter: Wang Qiang Priority: Critical Attachments: patch_20130425_01.txt, patch.txt it happened that a table with unbalanced regions, as follows in my cluster(the cluster has 20 regionservers, the table has 12 regions): http://hadoopdev19.cm6:60030/ 1 http://hadoopdev8.cm6:60030/ 2 http://hadoopdev17.cm6:60030/ 1 http://hadoopdev12.cm6:60030/ 1 http://hadoopdev5.cm6:60030/ 1 http://hadoopdev9.cm6:60030/ 1 http://hadoopdev22.cm6:60030/ 1 http://hadoopdev11.cm6:60030/ 1 http://hadoopdev21.cm6:60030/ 1 http://hadoopdev16.cm6:60030/ 1 http://hadoopdev10.cm6:60030/ 1 with the 'org.apache.hadoop.hbase.master.DefaultLoadBalancer', after 5 times load-balances, the table are still unbalanced: http://hadoopdev3.cm6:60030/ 1 http://hadoopdev20.cm6:60030/ 1 http://hadoopdev4.cm6:60030/ 2 http://hadoopdev18.cm6:60030/ 1 http://hadoopdev12.cm6:60030/ 1 http://hadoopdev14.cm6:60030/ 1 http://hadoopdev15.cm6:60030/ 1 http://hadoopdev6.cm6:60030/ 1 http://hadoopdev13.cm6:60030/ 1 http://hadoopdev11.cm6:60030/ 1 http://hadoopdev10.cm6:60030/ 1 http://hadoopdev19.cm6:60030/ 1 http://hadoopdev17.cm6:60030/ 1 http://hadoopdev8.cm6:60030/ 1 http://hadoopdev5.cm6:60030/ 1 http://hadoopdev12.cm6:60030/ 1 http://hadoopdev22.cm6:60030/ 1 http://hadoopdev11.cm6:60030/ 1 http://hadoopdev21.cm6:60030/ 1 http://hadoopdev7.cm6:60030/ 2 http://hadoopdev10.cm6:60030/ 1 http://hadoopdev16.cm6:60030/ 1 http://hadoopdev3.cm6:60030/ 1 http://hadoopdev20.cm6:60030/ 1 http://hadoopdev4.cm6:60030/ 1 http://hadoopdev18.cm6:60030/ 2 http://hadoopdev12.cm6:60030/ 1 http://hadoopdev14.cm6:60030/ 1 http://hadoopdev15.cm6:60030/ 1 http://hadoopdev6.cm6:60030/ 1 http://hadoopdev13.cm6:60030/ 1 http://hadoopdev11.cm6:60030/ 1 http://hadoopdev10.cm6:60030/ 1 http://hadoopdev19.cm6:60030/ 1 http://hadoopdev8.cm6:60030/ 1 http://hadoopdev17.cm6:60030/ 1 http://hadoopdev12.cm6:60030/ 1 http://hadoopdev5.cm6:60030/ 1 http://hadoopdev22.cm6:60030/ 1 http://hadoopdev11.cm6:60030/ 1 http://hadoopdev7.cm6:60030/ 1 http://hadoopdev21.cm6:60030/ 2 http://hadoopdev16.cm6:60030/ 1 http://hadoopdev10.cm6:60030/ 1 http://hadoopdev3.cm6:60030/ 1 http://hadoopdev20.cm6:60030/ 1 http://hadoopdev18.cm6:60030/ 1 http://hadoopdev4.cm6:60030/ 1 http://hadoopdev12.cm6:60030/ 1 http://hadoopdev15.cm6:60030/ 1 http://hadoopdev14.cm6:60030/ 2 http://hadoopdev6.cm6:60030/ 1 http://hadoopdev13.cm6:60030/ 1 http://hadoopdev11.cm6:60030/ 1 http://hadoopdev10.cm6:60030/ 1 from the above logs, we can also find that some regions needn't move, but they moved. follow into 'org.apache.hadoop.hbase.master.DefaultLoadBalancer.balanceCluster()', I found that 'maxToTake' is error calculated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8941) TestAccessController.testGlobalPermissionList failed with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708413#comment-13708413
]
Hudson commented on HBASE-8941:
---
FAILURE: Integrated in hbase-0.95-on-hadoop2 #179 (See
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/179/])
HBASE-8941 TestAccessController.testGlobalPermissionList failed with
IndexOutOfBoundsException (stack: rev 1503083)
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
TestAccessController.testGlobalPermissionList failed with
IndexOutOfBoundsException
---
Key: HBASE-8941
URL: https://issues.apache.org/jira/browse/HBASE-8941
Project: HBase
Issue Type: Test
Reporter: Jimmy Xiang
Assignee: stack
Fix For: 0.95.2
Attachments: 8941.txt, 8941v2.txt, 8941v3.txt
https://builds.apache.org/job/HBase-TRUNK/4246/testReport/junit/org.apache.hadoop.hbase.security.access/TestAccessController/testGlobalPermissionList/
{noformat}
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.hbase.security.access.TestAccessController.setUp(TestAccessController.java:188)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8941) TestAccessController.testGlobalPermissionList failed with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708416#comment-13708416
]
Hudson commented on HBASE-8941:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #616 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/616/])
HBASE-8941 TestAccessController.testGlobalPermissionList failed with
IndexOutOfBoundsException (stack: rev 1503082)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
TestAccessController.testGlobalPermissionList failed with
IndexOutOfBoundsException
---
Key: HBASE-8941
URL: https://issues.apache.org/jira/browse/HBASE-8941
Project: HBase
Issue Type: Test
Reporter: Jimmy Xiang
Assignee: stack
Fix For: 0.95.2
Attachments: 8941.txt, 8941v2.txt, 8941v3.txt
https://builds.apache.org/job/HBase-TRUNK/4246/testReport/junit/org.apache.hadoop.hbase.security.access/TestAccessController/testGlobalPermissionList/
{noformat}
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at
org.apache.hadoop.hbase.security.access.TestAccessController.setUp(TestAccessController.java:188)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere
[ https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708429#comment-13708429 ] Jean-Marc Spaggiari commented on HBASE-8927: Hi [~enis], as I just replied to [~saint@gmail.com], you're right. This was just to provide an example. They have nanoTime() all over the place, not just on this patch. But they also still have some currentMs() calls... Might be interesting to ask them ;) Use nano time instead of mili time everywhere - Key: HBASE-8927 URL: https://issues.apache.org/jira/browse/HBASE-8927 Project: HBase Issue Type: Bug Reporter: stack Attachments: 8927.txt Less collisions and we are paying the price of a long anyways so might as well fill it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708433#comment-13708433 ] Jean-Marc Spaggiari commented on HBASE-8755: Hi [~fenghh], the serves dedicated for those tests were to much different so I think it will have not been a good idea to run on them (difficult to interpret the results). So I just bought 3 absolutely identical nodes (the same as one I already have). By the end of the week I will have 4 servers with the same MB, same CPU, same memory (branch, PN, etc.) and same hard drive! Master will have a 1xSSD+1xSATA, others will have 2xSATA. To start. I will run the YCSB on that with and without this patch as soon as I get the hardware and I install the OS. Should be sometime later this week. I will also see if I can run PE. More to come. A new write thread model for HLog to improve the overall HBase write throughput --- Key: HBASE-8755 URL: https://issues.apache.org/jira/browse/HBASE-8755 Project: HBase Issue Type: Improvement Components: wal Reporter: Feng Honghua Attachments: HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-trunk-V0.patch In current write model, each write handler thread (executing put()) will individually go through a full 'append (hlog local buffer) = HLog writer append (write to hdfs) = HLog writer sync (sync hdfs)' cycle for each write, which incurs heavy race condition on updateLock and flushLock. The only optimization where checking if current syncTillHere txid in expectation for other thread help write/sync its own txid to hdfs and omitting the write/sync actually help much less than expectation. Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi proposed a new write thread model for writing hdfs sequence file and the prototype implementation shows a 4X improvement for throughput (from 17000 to 7+). I apply this new write thread model in HLog and the performance test in our test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) even beats the one of BigTable (Precolator published in 2011 says Bigtable's write throughput then is 31002). I can provide the detailed performance test results if anyone is interested. The change for new write thread model is as below: 1 All put handler threads append the edits to HLog's local pending buffer; (it notifies AsyncWriter thread that there is new edits in local buffer) 2 All put handler threads wait in HLog.syncer() function for underlying threads to finish the sync that contains its txid; 3 An single AsyncWriter thread is responsible for retrieve all the buffered edits in HLog's local pending buffer and write to the hdfs (hlog.writer.append); (it notifies AsyncFlusher thread that there is new writes to hdfs that needs a sync) 4 An single AsyncFlusher thread is responsible for issuing a sync to hdfs to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread that sync watermark increases) 5 An single AsyncNotifier thread is responsible for notifying all pending put handler threads which are waiting in the HLog.syncer() function 6 No LogSyncer thread any more (since there is always AsyncWriter/AsyncFlusher threads do the same job it does) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8348) Polish the migration to 0.96
[
https://issues.apache.org/jira/browse/HBASE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708438#comment-13708438
]
Hadoop QA commented on HBASE-8348:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592318/HBASE-8348_trunk_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6339//console
This message is automatically generated.
Polish the migration to 0.96
Key: HBASE-8348
URL: https://issues.apache.org/jira/browse/HBASE-8348
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0
Reporter: Jean-Daniel Cryans
Assignee: rajeshbabu
Priority: Critical
Fix For: 0.95.2
Attachments: HBASE-8348_trunk.patch, HBASE-8348_trunk_v2.patch
Currently, migration works but there's still a couple of rough edges:
- HBASE-8045 finished the .META. migration but didn't remove ROOT, so it's
still on the filesystem.
- Data in ZK needs to be removed manually. Either we fix up the data in ZK
or we delete it ourselves.
- TestMetaMigrationRemovingHTD has a testMetaUpdatedFlagInROOT method, but
ROOT is gone now.
Elliott was also mentioning that we could have hbase migrate do the HFileV1
checks, clear ZK, remove ROOT, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8874) PutCombiner is skipping KeyValues while combining puts of same row during bulkload
[
https://issues.apache.org/jira/browse/HBASE-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8874:
--
Status: Patch Available (was: Open)
PutCombiner is skipping KeyValues while combining puts of same row during
bulkload
--
Key: HBASE-8874
URL: https://issues.apache.org/jira/browse/HBASE-8874
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.95.1, 0.95.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: HBASE-8874_trunk.patch
While combining puts of same row in map phase we are using below logic in
PutCombiner#reduce. In for loop first time we will add one Put object to puts
map. Next time onwards we are just overriding key values of a family with key
values of the same family in other put. So we are mostly writing one Put
object to map output and remaining will be skipped(data loss).
{code}
Mapbyte[], Put puts = new TreeMapbyte[], Put(Bytes.BYTES_COMPARATOR);
for (Put p : vals) {
cnt++;
if (!puts.containsKey(p.getRow())) {
puts.put(p.getRow(), p);
} else {
puts.get(p.getRow()).getFamilyMap().putAll(p.getFamilyMap());
}
}
{code}
We need to change logic similar as below because we are sure the rowkey of
all the puts will be same.
{code}
Put finalPut = null;
Mapbyte[], List? extends Cell familyMap = null;
for (Put p : vals) {
cnt++;
if (finalPut==null) {
finalPut = p;
familyMap = finalPut.getFamilyMap();
} else {
for (Entrybyte[], List? extends Cell entry :
p.getFamilyMap().entrySet()) {
List? extends Cell list = familyMap.get(entry.getKey());
if (list == null) {
familyMap.put(entry.getKey(), entry.getValue());
} else {
(((ListKeyValue)list)).addAll((ListKeyValue)entry.getValue());
}
}
}
}
context.write(row, finalPut);
{code}
Also need to implement TODOs mentioned by Nick
{code}
// TODO: would be better if we knew codeK row/code and Put rowkey were
// identical. Then this whole Put buffering business goes away.
// TODO: Could use HeapSize to create an upper bound on the memory size of
// the puts map and flush some portion of the content while looping. This
// flush could result in multiple Puts for a single rowkey. That is
// acceptable because Combiner is run as an optimization and it's not
// critical that all Puts are grouped perfectly.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8874) PutCombiner is skipping KeyValues while combining puts of same row during bulkload
[
https://issues.apache.org/jira/browse/HBASE-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8874:
--
Attachment: HBASE-8874_trunk.patch
Patch for trunk. Verified the patch in the cluster
1) by taking same rowkey in multiple lines then also its giving all keyvalues.
2)configured putcombiner.row.threshold to small value and checked mapoutput
file which has multiple writes to same row.
Its working fine.
Please review it.
PutCombiner is skipping KeyValues while combining puts of same row during
bulkload
--
Key: HBASE-8874
URL: https://issues.apache.org/jira/browse/HBASE-8874
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.95.0, 0.95.1
Reporter: rajeshbabu
Assignee: rajeshbabu
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: HBASE-8874_trunk.patch
While combining puts of same row in map phase we are using below logic in
PutCombiner#reduce. In for loop first time we will add one Put object to puts
map. Next time onwards we are just overriding key values of a family with key
values of the same family in other put. So we are mostly writing one Put
object to map output and remaining will be skipped(data loss).
{code}
Mapbyte[], Put puts = new TreeMapbyte[], Put(Bytes.BYTES_COMPARATOR);
for (Put p : vals) {
cnt++;
if (!puts.containsKey(p.getRow())) {
puts.put(p.getRow(), p);
} else {
puts.get(p.getRow()).getFamilyMap().putAll(p.getFamilyMap());
}
}
{code}
We need to change logic similar as below because we are sure the rowkey of
all the puts will be same.
{code}
Put finalPut = null;
Mapbyte[], List? extends Cell familyMap = null;
for (Put p : vals) {
cnt++;
if (finalPut==null) {
finalPut = p;
familyMap = finalPut.getFamilyMap();
} else {
for (Entrybyte[], List? extends Cell entry :
p.getFamilyMap().entrySet()) {
List? extends Cell list = familyMap.get(entry.getKey());
if (list == null) {
familyMap.put(entry.getKey(), entry.getValue());
} else {
(((ListKeyValue)list)).addAll((ListKeyValue)entry.getValue());
}
}
}
}
context.write(row, finalPut);
{code}
Also need to implement TODOs mentioned by Nick
{code}
// TODO: would be better if we knew codeK row/code and Put rowkey were
// identical. Then this whole Put buffering business goes away.
// TODO: Could use HeapSize to create an upper bound on the memory size of
// the puts map and flush some portion of the content while looping. This
// flush could result in multiple Puts for a single rowkey. That is
// acceptable because Combiner is run as an optimization and it's not
// critical that all Puts are grouped perfectly.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708473#comment-13708473 ] Feng Honghua commented on HBASE-8755: - Thanks a lot [~jmspaggi], looking forward to your result. A new write thread model for HLog to improve the overall HBase write throughput --- Key: HBASE-8755 URL: https://issues.apache.org/jira/browse/HBASE-8755 Project: HBase Issue Type: Improvement Components: wal Reporter: Feng Honghua Attachments: HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-trunk-V0.patch In current write model, each write handler thread (executing put()) will individually go through a full 'append (hlog local buffer) = HLog writer append (write to hdfs) = HLog writer sync (sync hdfs)' cycle for each write, which incurs heavy race condition on updateLock and flushLock. The only optimization where checking if current syncTillHere txid in expectation for other thread help write/sync its own txid to hdfs and omitting the write/sync actually help much less than expectation. Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi proposed a new write thread model for writing hdfs sequence file and the prototype implementation shows a 4X improvement for throughput (from 17000 to 7+). I apply this new write thread model in HLog and the performance test in our test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) even beats the one of BigTable (Precolator published in 2011 says Bigtable's write throughput then is 31002). I can provide the detailed performance test results if anyone is interested. The change for new write thread model is as below: 1 All put handler threads append the edits to HLog's local pending buffer; (it notifies AsyncWriter thread that there is new edits in local buffer) 2 All put handler threads wait in HLog.syncer() function for underlying threads to finish the sync that contains its txid; 3 An single AsyncWriter thread is responsible for retrieve all the buffered edits in HLog's local pending buffer and write to the hdfs (hlog.writer.append); (it notifies AsyncFlusher thread that there is new writes to hdfs that needs a sync) 4 An single AsyncFlusher thread is responsible for issuing a sync to hdfs to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread that sync watermark increases) 5 An single AsyncNotifier thread is responsible for notifying all pending put handler threads which are waiting in the HLog.syncer() function 6 No LogSyncer thread any more (since there is always AsyncWriter/AsyncFlusher threads do the same job it does) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8803) region_mover.rb should move multiple regions at a time
[
https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708498#comment-13708498
]
Jean-Marc Spaggiari commented on HBASE-8803:
(Sorry, long comment because of all those logs)
When regions are unload from the server to be assigned somewhere else, threads
are limited to the number of servers. So if you have only 9 nodes on your
cluster, then your server can send only to 8 other nodes, and that'S why you
have a 8 limitation.
When regions are re-loaded from the other nodes to the server, then the only
limitation is the thread parameter. Below are some tests for this scenario.
{code}
13/07/15 09:38:39 INFO region_mover: Moving region
83a139f0629f2afbccc16f28c1fc5b07 (1 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
20a2b8e4d5346d05fc5659a04d5ccc2d (2 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
7b4dabe9857fdec833791bf70a6446a0 (3 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
7e1feaaff3c1be8e413072b4671245bf (4 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
e6d0569bd931317429cb5a130ea6a7f4 (5 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
d2190b8d656d6a813096b8d7b85f124c (6 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
1c024b8cabfea6dd735c6444cc7237a0 (7 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
0dfe044b3625984954505cfdf7adb32d (8 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
71c9d0ba64b16fb18fa7907227cb866a (9 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
1ccf69eb2e53c0decaf2dfa7aca86257 (10 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
8f5afbea06948fd57014bab1e9e07ea3 (11 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
0f08b0a2377ec8633259a84668ee98ea (12 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
e7b5a672b8d600e5d2b3021d6e1804eb (13 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
753afdbeae4cdd39032285ca635ea51a (14 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
4ea46ce099c17587b064c003280e130a (15 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
90cb0fc165da7cf677b8bfc4ec1850d4 (16 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
9b5e03d5d38166f9be2a8ecc9d393780 (17 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
e9638c105de9dab6a7f889ca0da80227 (18 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
60bfa08feb8d82ceed03b4dc56979d3f (19 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
a37988e554deff2b1a8c7477639f3872 (20 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
135966bd36f58329f532fdd8aa4b0bc7 (21 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
6d2ba19a71695ed8b97eac6f772c37b6 (22 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
8b8804d988d60da967fe5fd1cea44f96 (23 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
a133c11bc0b5e22d30e4d92e40e8adf8 (24 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
18c67cefc95a0484242b74df70160d3d (25 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
9ea96fe31480fededf978d59a9ae148d (26 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
af85ba099ccb1f51630ae4a806918150 (27 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
57a5bbe2c33b13ec6dc4e28c21324b45 (28 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
3a28d3a027385a65bcd001516bc8da5f (29 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
00fd5546feaeb573e7232825d4b2af88 (30 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
9988b51ab86393b007fec59cccb6cfec (31 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
06d051f424d2ac80fc896bad1fc70a37 (32 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52
[jira] [Commented] (HBASE-8874) PutCombiner is skipping KeyValues while combining puts of same row during bulkload
[
https://issues.apache.org/jira/browse/HBASE-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708501#comment-13708501
]
Hadoop QA commented on HBASE-8874:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592325/HBASE-8874_trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6340//console
This message is automatically generated.
PutCombiner is skipping KeyValues while combining puts of same row during
bulkload
--
Key: HBASE-8874
URL: https://issues.apache.org/jira/browse/HBASE-8874
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.95.0, 0.95.1
Reporter: rajeshbabu
Assignee: rajeshbabu
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: HBASE-8874_trunk.patch
While combining puts of same row in map phase we are using below logic in
PutCombiner#reduce. In for loop first time we will add one Put object to puts
map. Next time onwards we are just overriding key values of a family with key
values of the same family in other put. So we are mostly writing one Put
object to map output and remaining will be skipped(data loss).
{code}
Mapbyte[], Put puts = new TreeMapbyte[], Put(Bytes.BYTES_COMPARATOR);
for (Put p : vals) {
cnt++;
if (!puts.containsKey(p.getRow())) {
puts.put(p.getRow(), p);
} else {
puts.get(p.getRow()).getFamilyMap().putAll(p.getFamilyMap());
}
}
{code}
We need to change logic similar as below because we are sure the rowkey of
all the puts will be same.
{code}
Put finalPut = null;
Mapbyte[], List? extends Cell familyMap = null;
for (Put p : vals) {
cnt++;
if (finalPut==null) {
finalPut = p;
familyMap = finalPut.getFamilyMap();
} else {
for (Entrybyte[], List? extends Cell entry :
p.getFamilyMap().entrySet()) {
List? extends Cell list = familyMap.get(entry.getKey());
if (list == null) {
familyMap.put(entry.getKey(), entry.getValue());
[jira] [Commented] (HBASE-8432) a table with unbalanced regions will balance indefinitely with the 'org.apache.hadoop.hbase.master.DefaultLoadBalancer'
[
https://issues.apache.org/jira/browse/HBASE-8432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708506#comment-13708506
]
Ted Yu commented on HBASE-8432:
---
Second patch should have been taken at the root of workspace.
{code}
if(regionCount = max) {
break;
}
+BalanceInfo balanceInfo =
serverBalanceInfo.get(server.getKey().getServerName());
+if(balanceInfo != null) {
+ regionCount += balanceInfo.getNumRegionsAdded();
+}
+if(regionCount = max) {
+ continue;
+}
{code}
If accounting for BalanceInfo is moved to above 'if(regionCount = max)', we
can save one check for 'if(regionCount = max)'
Please attach trunk patch as well.
a table with unbalanced regions will balance indefinitely with the
'org.apache.hadoop.hbase.master.DefaultLoadBalancer'
---
Key: HBASE-8432
URL: https://issues.apache.org/jira/browse/HBASE-8432
Project: HBase
Issue Type: Bug
Components: Balancer
Affects Versions: 0.94.5
Environment: Linux 2.6.32-el5.x86_64
Reporter: Wang Qiang
Priority: Critical
Attachments: patch_20130425_01.txt, patch.txt
it happened that a table with unbalanced regions, as follows in my
cluster(the cluster has 20 regionservers, the table has 12 regions):
http://hadoopdev19.cm6:60030/ 1
http://hadoopdev8.cm6:60030/ 2
http://hadoopdev17.cm6:60030/ 1
http://hadoopdev12.cm6:60030/ 1
http://hadoopdev5.cm6:60030/ 1
http://hadoopdev9.cm6:60030/ 1
http://hadoopdev22.cm6:60030/ 1
http://hadoopdev11.cm6:60030/ 1
http://hadoopdev21.cm6:60030/ 1
http://hadoopdev16.cm6:60030/ 1
http://hadoopdev10.cm6:60030/ 1
with the 'org.apache.hadoop.hbase.master.DefaultLoadBalancer', after 5 times
load-balances, the table are still unbalanced:
http://hadoopdev3.cm6:60030/ 1
http://hadoopdev20.cm6:60030/ 1
http://hadoopdev4.cm6:60030/ 2
http://hadoopdev18.cm6:60030/ 1
http://hadoopdev12.cm6:60030/ 1
http://hadoopdev14.cm6:60030/ 1
http://hadoopdev15.cm6:60030/ 1
http://hadoopdev6.cm6:60030/ 1
http://hadoopdev13.cm6:60030/ 1
http://hadoopdev11.cm6:60030/ 1
http://hadoopdev10.cm6:60030/ 1
http://hadoopdev19.cm6:60030/ 1
http://hadoopdev17.cm6:60030/ 1
http://hadoopdev8.cm6:60030/ 1
http://hadoopdev5.cm6:60030/ 1
http://hadoopdev12.cm6:60030/ 1
http://hadoopdev22.cm6:60030/ 1
http://hadoopdev11.cm6:60030/ 1
http://hadoopdev21.cm6:60030/ 1
http://hadoopdev7.cm6:60030/ 2
http://hadoopdev10.cm6:60030/ 1
http://hadoopdev16.cm6:60030/ 1
http://hadoopdev3.cm6:60030/ 1
http://hadoopdev20.cm6:60030/ 1
http://hadoopdev4.cm6:60030/ 1
http://hadoopdev18.cm6:60030/ 2
http://hadoopdev12.cm6:60030/ 1
http://hadoopdev14.cm6:60030/ 1
http://hadoopdev15.cm6:60030/ 1
http://hadoopdev6.cm6:60030/ 1
http://hadoopdev13.cm6:60030/ 1
http://hadoopdev11.cm6:60030/ 1
http://hadoopdev10.cm6:60030/ 1
http://hadoopdev19.cm6:60030/ 1
http://hadoopdev8.cm6:60030/ 1
http://hadoopdev17.cm6:60030/ 1
http://hadoopdev12.cm6:60030/ 1
http://hadoopdev5.cm6:60030/ 1
http://hadoopdev22.cm6:60030/ 1
http://hadoopdev11.cm6:60030/ 1
http://hadoopdev7.cm6:60030/ 1
http://hadoopdev21.cm6:60030/ 2
http://hadoopdev16.cm6:60030/ 1
http://hadoopdev10.cm6:60030/ 1
http://hadoopdev3.cm6:60030/ 1
http://hadoopdev20.cm6:60030/ 1
http://hadoopdev18.cm6:60030/ 1
http://hadoopdev4.cm6:60030/ 1
http://hadoopdev12.cm6:60030/ 1
http://hadoopdev15.cm6:60030/ 1
http://hadoopdev14.cm6:60030/ 2
http://hadoopdev6.cm6:60030/ 1
http://hadoopdev13.cm6:60030/ 1
http://hadoopdev11.cm6:60030/ 1
http://hadoopdev10.cm6:60030/ 1
from the above logs, we can also find that some regions needn't move, but
they moved. follow into
'org.apache.hadoop.hbase.master.DefaultLoadBalancer.balanceCluster()', I
found that 'maxToTake' is error calculated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708525#comment-13708525
]
Lars Hofhansl commented on HBASE-8806:
--
Any chance to redo the perf test just for the case with no dups, just to be
sure?
(Or post the test code, and I'll do it)
I still have a slight preference for this patch (over HBASE-8877) for 0.94.
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8949) hbase.mapreduce.hfileoutputformat.blocksize should configure with blocksize of a table
rajeshbabu created HBASE-8949: - Summary: hbase.mapreduce.hfileoutputformat.blocksize should configure with blocksize of a table Key: HBASE-8949 URL: https://issues.apache.org/jira/browse/HBASE-8949 Project: HBase Issue Type: Bug Components: mapreduce Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0, 0.95.2, 0.94.10 While initializing mapreduce job we are not configuring hbase.mapreduce.hfileoutputformat.blocksize, so hfiles are always creating with 64kb (default)block size even though tables has different block size. We need to configure it with block size from table descriptor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8806:
---
Attachment: (was: hbase-8806-0.94-v6-microbenchmark)
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch,
hbase-8806-0.94-v6-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-v6-microbenchmark.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8806:
---
Attachment: hbase-8806-0.94-v6-microbenchmark
hbase-8806-0.94-v6-microbenchmark-no-dupe-rows
[~lhofhansl], as requested additional benchmark runs.
I've attached a couple additional runs of the benchmark against the current tip
of 0.94 and with the v6 patch applied.
In each you can see the first run was burnining in the VM but afterward the
variation in results gets much smaller.
In particular with no duplicate row keys, you can see the patch takes on
average 31-32ms to apply 25k puts in each mini batch. Without the patch it
seems to average 28-29ms. I'm comfortable with that difference given the
speedup in all other cases but I will run the numbers more carefully also for
the reentrant patch at HBSE-8877.
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch,
hbase-8806-0.94-v6-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-v6-microbenchmark.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8806:
---
Attachment: hbase-8806-0.94-v6-microbenchmark.txt
hbase-8806-0.94-v6-microbenchmark-no-dupe-rows.txt
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch,
hbase-8806-0.94-v6-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-v6-microbenchmark.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8806:
---
Attachment: (was: hbase-8806-0.94-v6-microbenchmark-no-dupe-rows)
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch,
hbase-8806-0.94-v6-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-v6-microbenchmark.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8908) Backport HBASE-8882 and HBASE-8904 (An Integration Test to Test MTTR) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708585#comment-13708585 ] Elliott Clark commented on HBASE-8908: -- +1 lgtm Backport HBASE-8882 and HBASE-8904 (An Integration Test to Test MTTR) to 0.94 - Key: HBASE-8908 URL: https://issues.apache.org/jira/browse/HBASE-8908 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.10 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: 8908.patch, 8908.patch, 8908.patch, IntegrationTestMTTR-output.txt Easy low risk backport. See how 0.94 stacks up against later versions with respect to MTTR, and vice versa. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8806:
---
Attachment: (was: hbase-8806-0.94-v6-microbenchmark.txt)
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch, hbase-8806-0.94-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-microbenchmarks-dupe-rows.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8806:
---
Attachment: (was: hbase-8806-0.94-v6-microbenchmark-no-dupe-rows.txt)
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch, hbase-8806-0.94-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-microbenchmarks-dupe-rows.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8806:
---
Attachment: hbase-8806-0.94-microbenchmark-no-dupe-rows.txt
hbase-8806-0.94-microbenchmarks-dupe-rows.txt
Updated the benchmarks to also compare HBASE-8877-0.94-v2.patch.
So two files, each one compares current 0.94 versus 8806-0.94-v6.txt versus
HBASE-8877-0.94-v2.patch. One file varies the proportion of duplicate rows the
other runs several sets of trials with no duplicate rows.
I'd say the difference between the 3 is not significant for the case of no
duplicate rows, though 8806-0.94-v6.txt may be slightly slower.
I'm happy to see either 8806-0.94-v6.txt or HBASE-8877-0.94-v2.patch go in for
0.94
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch, hbase-8806-0.94-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-microbenchmarks-dupe-rows.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8908) Backport HBASE-8882 and HBASE-8904 (An Integration Test to Test MTTR) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708598#comment-13708598 ] Andrew Purtell commented on HBASE-8908: --- Thanks for the great test and for taking a look here too [~eclark]. Going to commit this evening unless objection. Backport HBASE-8882 and HBASE-8904 (An Integration Test to Test MTTR) to 0.94 - Key: HBASE-8908 URL: https://issues.apache.org/jira/browse/HBASE-8908 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.10 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: 8908.patch, 8908.patch, 8908.patch, IntegrationTestMTTR-output.txt Easy low risk backport. See how 0.94 stacks up against later versions with respect to MTTR, and vice versa. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708607#comment-13708607
]
Hadoop QA commented on HBASE-8806:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592344/hbase-8806-0.94-microbenchmark-no-dupe-rows.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6342//console
This message is automatically generated.
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch, hbase-8806-0.94-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-microbenchmarks-dupe-rows.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708587#comment-13708587
]
Hadoop QA commented on HBASE-8806:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592342/hbase-8806-0.94-v6-microbenchmark.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6341//console
This message is automatically generated.
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch,
hbase-8806-0.94-v6-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-v6-microbenchmark.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8950) Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
[
https://issues.apache.org/jira/browse/HBASE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Williams updated HBASE-8950:
--
Description:
Currently, the HbaseTestingUtility is not importable in versions 0.94.x without
Sonatype Aether dynamic classpath modification. This is clunky and makes
importing the utility opaque and unfriendly. Please change this to make the
HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
was:Currently, the HbaseTestingUtility is not importable in versions 0.94.x
without Sonatype Aether dynamic classpath modification. This is clunky and
makes importing the utility opaque and unfriendly. Please change this to make
the HbaseTestingUtility visible and importable without special classpath
modifications.
Summary: Please Make HbaseTestingUtility Import without Dynamic Class
Path Modification (was: Please Make HbaseTestingUtility Import without Dynamic
Class Path Mods)
Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
--
Key: HBASE-8950
URL: https://issues.apache.org/jira/browse/HBASE-8950
Project: HBase
Issue Type: Improvement
Reporter: David Williams
Currently, the HbaseTestingUtility is not importable in versions 0.94.x
without Sonatype Aether dynamic classpath modification. This is clunky and
makes importing the utility opaque and unfriendly. Please change this to
make the HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8950) Please Make HbaseTestingUtility Import without Dynamic Class Path Mods
David Williams created HBASE-8950: - Summary: Please Make HbaseTestingUtility Import without Dynamic Class Path Mods Key: HBASE-8950 URL: https://issues.apache.org/jira/browse/HBASE-8950 Project: HBase Issue Type: Improvement Reporter: David Williams Currently, the HbaseTestingUtility is not importable in versions 0.94.x without Sonatype Aether dynamic classpath modification. This is clunky and makes importing the utility opaque and unfriendly. Please change this to make the HbaseTestingUtility visible and importable without special classpath modifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8950) Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
[
https://issues.apache.org/jira/browse/HBASE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Williams updated HBASE-8950:
--
Description:
Currently, the HbaseTestingUtility is not importable in versions 0.94.x without
Sonatype Aether dynamic classpath modification. This is clunky and makes
importing the utility opaque and unfriendly. Please change this to make the
HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on the
classpath.
was:
Currently, the HbaseTestingUtility is not importable in versions 0.94.x without
Sonatype Aether dynamic classpath modification. This is clunky and makes
importing the utility opaque and unfriendly. Please change this to make the
HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
--
Key: HBASE-8950
URL: https://issues.apache.org/jira/browse/HBASE-8950
Project: HBase
Issue Type: Improvement
Reporter: David Williams
Currently, the HbaseTestingUtility is not importable in versions 0.94.x
without Sonatype Aether dynamic classpath modification. This is clunky and
makes importing the utility opaque and unfriendly. Please change this to
make the HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on
the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8877) Reentrant row locks
[
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8877:
---
Attachment: HBASE-8877-v7.patch
Thanks, Stack, for the review. Judging by your comments I believe you're
addressing HBASE-8877-v6.patch (rather than HBASE-8877-0.94-v2.patch).
{quote}A bit of doc on this would help: ... rowLocksHeldByThread{quote}
Done.
{quote}hould we deprecate stuff like this, public OperationStatus[] put(Put[]
puts) throws IOException {, the methods that you have made into pass-throughs?
(Fine in another issue){quote}
That's the only method that became a pass-through. I considered just removing
it - we're already changing the HRegion public methods from row locks - I
assumed that was OK to do in the singularity (not to mention the CP change).
It's only used by HBaseFsck and 3 tests. I've now inlined it directly. Let me
know if you think it needs to be deprecated for a release first.
{quote}When we fall out here because we failed to acquire a lock, what happens?
We apply all mutations for which we did get a lock? And then return the client
reporting as failed those we did not get a lock on?{quote}
If we did get any locks then we apply those mutations for which we did get
locks, clean them up, return out to batchMutate which will continue again. If
we failed to get any locks at all, then it means the timeout expired (default
30 seconds) and by my reading it still goes through all the cleanup methods
then tries again. None of that logic was touched for these issues.
{quote}Is this racey?{quote}
Strictly rated G
{quote}Could another thread come in and add to lockedRows before get to add
this rowKey to the thread local?{quote}
Nope, lockedRows.putIfAbsent gives atomicity so the next thread would get back
a non null existingContext.
{quote}This should be finals{quote}
Agreed. Done.
Attached HBASE-8877-v7.patch
Reentrant row locks
---
Key: HBASE-8877
URL: https://issues.apache.org/jira/browse/HBASE-8877
Project: HBase
Issue Type: Bug
Components: Coprocessors, regionserver
Reporter: Dave Latham
Assignee: Dave Latham
Fix For: 0.95.2
Attachments: hbase-8877-0.94-microbenchmark.txt,
HBASE-8877-0.94.patch, HBASE-8877-0.94-v2.patch, HBASE-8877.patch,
HBASE-8877-v2.patch, HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt,
HBASE-8877-v4.patch, HBASE-8877-v5.patch, HBASE-8877-v6.patch,
HBASE-8877-v7.patch
HBASE-8806 revealed performance problems with batch mutations failing to
reacquire the same row locks. It looks like HBASE-8806 will use a less
intrusive change for 0.94 to have batch mutations track their own row locks
and not attempt to reacquire them. Another approach will be to support
reentrant row locks directly. This allows simplifying a great deal of
calling code to no longer track and pass around lock ids.
One affect this change will have is changing the RegionObserver coprocessor's
methods preBatchMutate and postBatchMutate from taking a
{{MiniBatchOperationInProgressPairMutation, Integer miniBatchOp}} to
taking a {{MiniBatchOperationInProgressMutation miniBatchOp}}. I don't
believe CPs should be relying on these lock ids, but that's a potential
incompatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8496) Implement tags and the internals of how a tag should look like
[ https://issues.apache.org/jira/browse/HBASE-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708649#comment-13708649 ] Andrew Purtell commented on HBASE-8496: --- bq. So HFileV3 would be the one which would know about the Tags and the read and write path in HFileV3 would understand tags. What I especially like about this is any performance or functional risks which might possibly be introduced by the tags work is completely optional to take on this way - just don't select HFileV3. We can even mark it as experimental until 0.98. I have been able to observe Ram's work over the past couple of months trying out various other approaches just shy of introducing a new file format. It's not a decision come to overnight. +1 Implement tags and the internals of how a tag should look like -- Key: HBASE-8496 URL: https://issues.apache.org/jira/browse/HBASE-8496 Project: HBase Issue Type: New Feature Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: Tag design.pdf The intent of this JIRA comes from HBASE-7897. This would help us to decide on the structure and format of how the tags should look like. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8877) Reentrant row locks
[
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708660#comment-13708660
]
Dave Latham commented on HBASE-8877:
3 +1's on the dev mailing list for the removal of lock ids from the
coprocessor, in addition to Andrew's +1 here. No -1's.
One thought - should we commit to 0.94 a javadoc method at least warning that
the locks are going to disappear from the coprocessor?
Reentrant row locks
---
Key: HBASE-8877
URL: https://issues.apache.org/jira/browse/HBASE-8877
Project: HBase
Issue Type: Bug
Components: Coprocessors, regionserver
Reporter: Dave Latham
Assignee: Dave Latham
Fix For: 0.95.2
Attachments: hbase-8877-0.94-microbenchmark.txt,
HBASE-8877-0.94.patch, HBASE-8877-0.94-v2.patch, HBASE-8877.patch,
HBASE-8877-v2.patch, HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt,
HBASE-8877-v4.patch, HBASE-8877-v5.patch, HBASE-8877-v6.patch,
HBASE-8877-v7.patch
HBASE-8806 revealed performance problems with batch mutations failing to
reacquire the same row locks. It looks like HBASE-8806 will use a less
intrusive change for 0.94 to have batch mutations track their own row locks
and not attempt to reacquire them. Another approach will be to support
reentrant row locks directly. This allows simplifying a great deal of
calling code to no longer track and pass around lock ids.
One affect this change will have is changing the RegionObserver coprocessor's
methods preBatchMutate and postBatchMutate from taking a
{{MiniBatchOperationInProgressPairMutation, Integer miniBatchOp}} to
taking a {{MiniBatchOperationInProgressMutation miniBatchOp}}. I don't
believe CPs should be relying on these lock ids, but that's a potential
incompatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8662) [rest] support impersonation
[ https://issues.apache.org/jira/browse/HBASE-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708685#comment-13708685 ] Hudson commented on HBASE-8662: --- ABORTED: Integrated in HBase-TRUNK #4253 (See [https://builds.apache.org/job/HBase-TRUNK/4253/]) HBASE-8662 [rest] support impersonation (jxiang: rev 1503337) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/RPC.proto * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/Constants.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServlet.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServletContainer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/filter/AuthFilter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * /hbase/trunk/pom.xml [rest] support impersonation Key: HBASE-8662 URL: https://issues.apache.org/jira/browse/HBASE-8662 Project: HBase Issue Type: Sub-task Components: REST, security Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.98.0 Attachments: method_doas.patch, secure_rest.patch, trunk-8662.patch, trunk-8662_v2.patch, trunk-8662_v3.patch, trunk-8662_v4.patch, trunk-8662_v5.patch, trunk-8662_v6.patch, trunk-8662_v7.1.patch, trunk-8662_v7.2.patch, trunk-8662_v7.3.patch, trunk-8662_v7.patch Currently, our client API uses a fixed user: the current user. It should accept a user passed in, if authenticated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708695#comment-13708695
]
stack commented on HBASE-8944:
--
[~mobiusinversion] What hdfs/hadoop are you running? Looking at the hdfs src,
cut and paste what is at #426 in
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes if you don't mind.
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7875) introduce a compaction switch in HBase Shell
[
https://issues.apache.org/jira/browse/HBASE-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708700#comment-13708700
]
Sergey Shelukhin commented on HBASE-7875:
-
tests failed...
introduce a compaction switch in HBase Shell
Key: HBASE-7875
URL: https://issues.apache.org/jira/browse/HBASE-7875
Project: HBase
Issue Type: New Feature
Components: shell
Affects Versions: 0.95.2
Reporter: Liang Xie
Assignee: Liang Xie
Priority: Trivial
Attachments: HBASE-7875.txt, HBASE-7875-v1.patch,
HBASE-7875-v1.patch, HBase-7875-v2.txt
A trivial patch to add an per-table compaction switch. it would be helpful
for OPS or manual testing in real cluster, etc.
Usage:
hbase(main):016:0 alter 'YCSBTest', {METHOD = 'table_att',
COMPACTION_ENABLE = 'FALSE'}
Updating all regions with the new schema...
64/64 regions updated.
Done.
0 row(s) in 1.1290 seconds
hbase(main):017:0 describe 'YCSBTest'
DESCRIPTION
ENABLED
{NAME = 'YCSBTest', COMPACTION_ENABLE = 'false', SPLIT_POLICY =
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPoli true
cy', FAMILIES = [{NAME = 'test', DATA_BLOCK_ENCODING = 'NONE',
BLOOMFILTER = 'NONE', REPLICATION_SCOPE = '0', COMPRESSION = 'S
NAPPY', VERSIONS = '1', TTL = '2147483647', MIN_VERSIONS = '0',
KEEP_DELETED_CELLS = 'false', BLOCKSIZE = '65536', ENCODE_ON_DI
SK = 'true', IN_MEMORY = 'false', BLOCKCACHE = 'true'}]}
1 row(s) in 0.0210 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8950) Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
[
https://issues.apache.org/jira/browse/HBASE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708701#comment-13708701
]
stack commented on HBASE-8950:
--
bq. ...without Sonatype Aether dynamic classpath modification.
What is this? What does it do?
What is your dev context [~mobiusinversion]?
There is section in refguide on ides that may be of help (if you are using an
ide): http://hbase.apache.org/book.html#ides
Yeah, we're a maven'ized project so it can be a PITA getting all you need up on
the CLASSPATH.
Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
--
Key: HBASE-8950
URL: https://issues.apache.org/jira/browse/HBASE-8950
Project: HBase
Issue Type: Improvement
Reporter: David Williams
Currently, the HbaseTestingUtility is not importable in versions 0.94.x
without Sonatype Aether dynamic classpath modification. This is clunky and
makes importing the utility opaque and unfriendly. Please change this to
make the HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on
the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere
[ https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708733#comment-13708733 ] Sergey Shelukhin commented on HBASE-8927: - As I said can we please look at reasons to do this, regardless of what this clock does... there's not a single good reason to use nano in most places, with the exception of column timestamp precision, for which a more complex and safer scheme can be made as suggested by some comments above, and measuring time taken for short (normally expected to take single seconds or less) operations. Now, we could do it everywhere anyway just because why not, if it were free of other consequences, but in this case we want to ignore explicitly missing guarantees on the interface and rely on internals of particular JVM(s) and OS(es). If some JVM, OS, or future changes to JVM code change the implementation (still staying within the interface guarantees), there will be a fail of epic proportions. In my mind there isn't even a reason to look at internals and ponder, simply because we don't stand to gain anything from nano in most places. Use nano time instead of mili time everywhere - Key: HBASE-8927 URL: https://issues.apache.org/jira/browse/HBASE-8927 Project: HBase Issue Type: Bug Reporter: stack Attachments: 8927.txt Less collisions and we are paying the price of a long anyways so might as well fill it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708743#comment-13708743
]
David Williams commented on HBASE-8944:
---
Hi Stack, I'm checking out the source tree now. The sources aren't included on
Maven Central. Should know more when its down checking out. I am going to need
to identify which branch the 1.2.0 build is based on.
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8909) HBASE-6170 broke hbase.regionserver.lease.period config, we should support it for BC
[ https://issues.apache.org/jira/browse/HBASE-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8909: - Attachment: hbase-8909_v3.patch Added test category. Retesting with hadoopqa. HBASE-6170 broke hbase.regionserver.lease.period config, we should support it for BC -- Key: HBASE-8909 URL: https://issues.apache.org/jira/browse/HBASE-8909 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: hbase-8909_v1.patch, hbase-8909_v2.patch, hbase-8909_v3.patch HBASE-6170 replaces hbase.regionserver.lease.period with the new configuration hbase.client.scanner.timeout.period. We should support the old configuration at least in 0.96. We can put a WARN and deprecate the old value. Remove it in 1.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8950) Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
[
https://issues.apache.org/jira/browse/HBASE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708745#comment-13708745
]
Andrew Purtell commented on HBASE-8950:
---
What Stack said.
HBaseTestingUtility, and all code under test/, is targeted for HBase
contributors, not general test usage. Your mileage may vary. That said, maybe
we can help you, can you provide more detail on what you would like to see?
Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
--
Key: HBASE-8950
URL: https://issues.apache.org/jira/browse/HBASE-8950
Project: HBase
Issue Type: Improvement
Reporter: David Williams
Currently, the HbaseTestingUtility is not importable in versions 0.94.x
without Sonatype Aether dynamic classpath modification. This is clunky and
makes importing the utility opaque and unfriendly. Please change this to
make the HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on
the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708748#comment-13708748
]
stack commented on HBASE-8944:
--
[~mobiusinversion] So a 0.94.6.1 against an hadoop 1.2? If so, you are into
new territory so you very well may have uncovered an incompatibility. Lets
figure it. Thanks.
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8877) Reentrant row locks
[
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708754#comment-13708754
]
stack commented on HBASE-8877:
--
+1 on patch. Will commit to trunk in few hours unless objection.
No harm adding warning on 0.94 patch.
Reentrant row locks
---
Key: HBASE-8877
URL: https://issues.apache.org/jira/browse/HBASE-8877
Project: HBase
Issue Type: Bug
Components: Coprocessors, regionserver
Reporter: Dave Latham
Assignee: Dave Latham
Fix For: 0.95.2
Attachments: hbase-8877-0.94-microbenchmark.txt,
HBASE-8877-0.94.patch, HBASE-8877-0.94-v2.patch, HBASE-8877.patch,
HBASE-8877-v2.patch, HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt,
HBASE-8877-v4.patch, HBASE-8877-v5.patch, HBASE-8877-v6.patch,
HBASE-8877-v7.patch
HBASE-8806 revealed performance problems with batch mutations failing to
reacquire the same row locks. It looks like HBASE-8806 will use a less
intrusive change for 0.94 to have batch mutations track their own row locks
and not attempt to reacquire them. Another approach will be to support
reentrant row locks directly. This allows simplifying a great deal of
calling code to no longer track and pass around lock ids.
One affect this change will have is changing the RegionObserver coprocessor's
methods preBatchMutate and postBatchMutate from taking a
{{MiniBatchOperationInProgressPairMutation, Integer miniBatchOp}} to
taking a {{MiniBatchOperationInProgressMutation miniBatchOp}}. I don't
believe CPs should be relying on these lock ids, but that's a potential
incompatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708760#comment-13708760
]
Andrew Purtell commented on HBASE-8944:
---
Isn't this HDFS-2556? Try {{umask 022}} before running tests to confirm.
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Williams updated HBASE-8944:
--
Description:
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not in
certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
was:
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not in
certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708783#comment-13708783
]
David Williams commented on HBASE-8944:
---
Great, sounds good. I testes 0.94.6.1 against Hadoop 1.1.2 and 1.0.4. Using
1.1.2 the tests pass on Mac OSX but not CentOS. Using 1.0.4 the tests fail as
well on Mac OSX with this error:
{code}
ERROR in (create-table) (MiniHBaseCluster.java:224)
Uncaught exception, not in assertion.
expected: nil
actual: java.io.IOException: Shutting down
at org.apache.hadoop.hbase.MiniHBaseCluster.init (MiniHBaseCluster.java:224)
org.apache.hadoop.hbase.MiniHBaseCluster.init (MiniHBaseCluster.java:87)
org.apache.hadoop.hbase.MiniHBaseCluster.init (MiniHBaseCluster.java:78)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster
(HBaseTestingUtility.java:657)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:627)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:575)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:562)
hbase.table_test$test_config.doInvoke (table_test.clj:10)
clojure.lang.RestFn.invoke (RestFn.java:397)
hbase.table_test/fn (table_test.clj:19)
clojure.test$test_var$fn__7145.invoke (test.clj:701)
{code}
So so far the results look something like this:
||hbase version||hadoop version||Mac OSX||CentOS||
|0.94.9|1.2.0|pass|fail|
|0.94.9|1.1.2|pass|fail|
|0.94.6.1|1.2.0|pass|fail|
|0.94.6.1|1.1.2|pass|fail|
|0.94.6.1|1.0.4|fail|fail|
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8951) DNS not resolving should no longer show in logs as ERROR
Ron Echeverri created HBASE-8951:
Summary: DNS not resolving should no longer show in logs as ERROR
Key: HBASE-8951
URL: https://issues.apache.org/jira/browse/HBASE-8951
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: Ron Echeverri
Priority: Minor
Current error message from an hbase-0.94.5 installation:
{{ERROR mapreduce.TableInputFormatBase: Cannot resolve the host name for
/192.168.254.178 because of javax.naming.CommunicationException: DNS error
[Root exception is java.net.PortUnreachableException: ICMP Port Unreachable];
remaining name '178.254.168.192.in-addr.arpa'}}
According to the Book, Both forward and reverse DNS resolving must work in
versions of HBase previous to 0.92.0. Given that since 0.92.0,
forward/reverse DNS is not required, this should be a WARN and not an ERROR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708785#comment-13708785
]
David Williams commented on HBASE-8944:
---
Update, based on Andrew Purtell's input, running
{code}
$ umask 022
{code}
Fixes this issue on Linux
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708787#comment-13708787
]
David Williams commented on HBASE-8944:
---
Also for all version combinations above.
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8877) Reentrant row locks
[
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708794#comment-13708794
]
Hadoop QA commented on HBASE-8877:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592355/HBASE-8877-v7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestFullLogReconstruction
org.apache.hadoop.hbase.security.access.TestAccessController
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6343//console
This message is automatically generated.
Reentrant row locks
---
Key: HBASE-8877
URL: https://issues.apache.org/jira/browse/HBASE-8877
Project: HBase
Issue Type: Bug
Components: Coprocessors, regionserver
Reporter: Dave Latham
Assignee: Dave Latham
Fix For: 0.95.2
Attachments: hbase-8877-0.94-microbenchmark.txt,
HBASE-8877-0.94.patch, HBASE-8877-0.94-v2.patch, HBASE-8877.patch,
HBASE-8877-v2.patch, HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt,
HBASE-8877-v4.patch, HBASE-8877-v5.patch, HBASE-8877-v6.patch,
HBASE-8877-v7.patch
HBASE-8806 revealed performance problems with batch mutations failing to
reacquire the same row locks. It looks like HBASE-8806 will use a less
intrusive change for 0.94 to have batch mutations track their own row locks
and not attempt to reacquire them. Another approach will be to support
reentrant row locks directly. This allows simplifying a great deal of
calling code to no longer track and pass around lock ids.
One affect this change will have is changing the RegionObserver coprocessor's
methods preBatchMutate and postBatchMutate from taking a
{{MiniBatchOperationInProgressPairMutation, Integer miniBatchOp}} to
taking a {{MiniBatchOperationInProgressMutation miniBatchOp}}. I don't
believe CPs should be relying on these lock ids, but that's a potential
incompatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8950) Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
[
https://issues.apache.org/jira/browse/HBASE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Williams updated HBASE-8950:
--
Description:
Currently, the HbaseTestingUtility is not importable in versions 0.94.x without
Sonatype Aether dynamic classpath modification. Please change this to make the
HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on the
classpath.
was:
Currently, the HbaseTestingUtility is not importable in versions 0.94.x without
Sonatype Aether dynamic classpath modification. This is clunky and makes
importing the utility opaque and unfriendly. Please change this to make the
HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on the
classpath.
Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
--
Key: HBASE-8950
URL: https://issues.apache.org/jira/browse/HBASE-8950
Project: HBase
Issue Type: Improvement
Reporter: David Williams
Currently, the HbaseTestingUtility is not importable in versions 0.94.x
without Sonatype Aether dynamic classpath modification. Please change this to
make the HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on
the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8950) Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
[
https://issues.apache.org/jira/browse/HBASE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708797#comment-13708797
]
David Williams commented on HBASE-8950:
---
Hi Stack / Andrew,
On deeper inspection, this is specific to Leiningen and Clojure. As it stands,
importing the HBaseTestingUtility fails in Clojure unless a special flag is
passed to the Leiningen Pomegranate library.
https://github.com/cemerick/pomegranate
Andrew, thats good to know, the testing utility appears useful for lightweight
testing. Is there a better or more recommended way to spin up a temporary
cluster for unit testing?
Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
--
Key: HBASE-8950
URL: https://issues.apache.org/jira/browse/HBASE-8950
Project: HBase
Issue Type: Improvement
Reporter: David Williams
Currently, the HbaseTestingUtility is not importable in versions 0.94.x
without Sonatype Aether dynamic classpath modification. This is clunky and
makes importing the utility opaque and unfriendly. Please change this to
make the HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on
the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8877) Reentrant row locks
[
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708803#comment-13708803
]
Lars Hofhansl commented on HBASE-8877:
--
Do we need to do reference counting on the locks?
Technically the patch is correct, since an HBase lock does not outlive an RPC
request, but it still seems a tad fragile.
For example what if in the future a method does its own locking: It acquires a
lock on a row, does some logic, releases the lock. If a method higher up in the
call stack had the row locked, it is now unlocked from under it.
Reentrant row locks
---
Key: HBASE-8877
URL: https://issues.apache.org/jira/browse/HBASE-8877
Project: HBase
Issue Type: Bug
Components: Coprocessors, regionserver
Reporter: Dave Latham
Assignee: Dave Latham
Fix For: 0.95.2
Attachments: hbase-8877-0.94-microbenchmark.txt,
HBASE-8877-0.94.patch, HBASE-8877-0.94-v2.patch, HBASE-8877.patch,
HBASE-8877-v2.patch, HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt,
HBASE-8877-v4.patch, HBASE-8877-v5.patch, HBASE-8877-v6.patch,
HBASE-8877-v7.patch
HBASE-8806 revealed performance problems with batch mutations failing to
reacquire the same row locks. It looks like HBASE-8806 will use a less
intrusive change for 0.94 to have batch mutations track their own row locks
and not attempt to reacquire them. Another approach will be to support
reentrant row locks directly. This allows simplifying a great deal of
calling code to no longer track and pass around lock ids.
One affect this change will have is changing the RegionObserver coprocessor's
methods preBatchMutate and postBatchMutate from taking a
{{MiniBatchOperationInProgressPairMutation, Integer miniBatchOp}} to
taking a {{MiniBatchOperationInProgressMutation miniBatchOp}}. I don't
believe CPs should be relying on these lock ids, but that's a potential
incompatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reassigned HBASE-8944:
-
Assignee: Andrew Purtell
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
Assignee: Andrew Purtell
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-8944:
--
Attachment: 8944.patch
Thanks for taking the time to check [~mobiusinversion]. As this is HDFS-2556,
would you consider the attached update to our troubleshooting manual section
sufficient here?
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
Assignee: Andrew Purtell
Attachments: 8944.patch
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8950) Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
[
https://issues.apache.org/jira/browse/HBASE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708805#comment-13708805
]
stack commented on HBASE-8950:
--
So, I can resolve this issue then [~mobiusinversion] HBaseTestingUtility is
probably best tool but as Andrew says, it can be ornery getting it going.
There is also the https://github.com/kijiproject/fake-hbase project by our
brothers and sisters over at kiji.
Please Make HbaseTestingUtility Import without Dynamic Class Path Modification
--
Key: HBASE-8950
URL: https://issues.apache.org/jira/browse/HBASE-8950
Project: HBase
Issue Type: Improvement
Reporter: David Williams
Currently, the HbaseTestingUtility is not importable in versions 0.94.x
without Sonatype Aether dynamic classpath modification. Please change this to
make the HbaseTestingUtility visible and importable without special classpath
modifications.
That is:
{code}
import org.apache.hadoop.hbase.HBaseTestingUtility;
{code}
should succeed after placing org.apache.hbase/hbase 0.94.X and higher on
the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708817#comment-13708817
]
stack commented on HBASE-8944:
--
+1 on patch [~andrew.purt...@gmail.com]
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
Assignee: Andrew Purtell
Attachments: 8944.patch
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708814#comment-13708814 ] Jesse Yates commented on HBASE-8884: this is kind of a big patch... mind putting it up on RB? It would be good to get some other eyes on it (maybe [~andrew.purt...@gmail.com]?). Good stuff so far from my first pass though :) Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8942) DFS errors during a read operation (get/scan), may cause write outliers
[ https://issues.apache.org/jira/browse/HBASE-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer resolved HBASE-8942. Resolution: Fixed Fix Version/s: 0.89-fb DFS errors during a read operation (get/scan), may cause write outliers --- Key: HBASE-8942 URL: https://issues.apache.org/jira/browse/HBASE-8942 Project: HBase Issue Type: Bug Affects Versions: 0.89-fb, 0.95.2 Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Priority: Minor Fix For: 0.89-fb This is a similar issue as discussed in HBASE-8228 1) A scanner holds the Store.ReadLock() while opening the store files ... encounters errors. Thus, takes a long time to finish. 2) A flush is completed, in the mean while. It needs the write lock to commit(), and update scanners. Hence ends up waiting. 3+) All Puts (and also Gets) to the CF, which will need a read lock, will have to wait for 1) and 2) to complete. Thus blocking updates to the system for the DFS timeout. Fix: Open Store files outside the read lock. getScanners() already tries to do this optimisation. However, Store.getScanner() which calls this functions through the StoreScanner constructor, redundantly tries to grab the readLock. Causing the readLock to be held while the storeFiles are being opened, and seeked. We should get rid of the readLock() in Store.getScanner(). This is not required. The constructor for StoreScanner calls getScanners(xxx, xxx, xxx). This has the required locking already. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708811#comment-13708811
]
David Williams commented on HBASE-8944:
---
Hi Andrew, yes, absolutely. And thanks for taking the time to check this out.
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
Assignee: Andrew Purtell
Attachments: 8944.patch
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8911) Inject MTTR specific traces to get a break up of various steps
[ https://issues.apache.org/jira/browse/HBASE-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8911: --- Attachment: 8911-v2.patch Path that incorporate Elliott's reviews. Also, each of the class involved in fail-over has a separate Span now. Inject MTTR specific traces to get a break up of various steps -- Key: HBASE-8911 URL: https://issues.apache.org/jira/browse/HBASE-8911 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Attachments: 8911-v0.patch, 8911-v1.patch, 8911-v2.patch There are various steps involved in a regionserver recovery process. This jira adds instrumentation at various places in order to get an idea what are the steps involved in a regionserver recovery and how much time is spent in each of these parts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8806) Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for duplicate rows.
[
https://issues.apache.org/jira/browse/HBASE-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708835#comment-13708835
]
Lars Hofhansl commented on HBASE-8806:
--
Thanks [~davelatham]!
I would like to commit this patch to 0.94. I think the perf impact is an
acceptable trade off.
We can undo this if/when we apply HBASE-8877 to 0.94 (also see my question
about reference counting there).
Row locks are acquired repeatedly in HRegion.doMiniBatchMutation for
duplicate rows.
Key: HBASE-8806
URL: https://issues.apache.org/jira/browse/HBASE-8806
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.5
Reporter: rahul gidwani
Priority: Critical
Fix For: 0.98.0, 0.95.2, 0.94.10
Attachments: 8806-0.94-v4.txt, 8806-0.94-v5.txt, 8806-0.94-v6.txt,
HBASE-8806-0.94.10.patch, HBASE-8806-0.94.10-v2.patch,
HBASE-8806-0.94.10-v3.patch, hbase-8806-0.94-microbenchmark-no-dupe-rows.txt,
hbase-8806-0.94-microbenchmarks-dupe-rows.txt, HBASE-8806.patch,
HBASE-8806-threadBasedRowLocks.patch,
HBASE-8806-threadBasedRowLocks-v2.patch, row_lock_perf_results.txt
If we already have the lock in the doMiniBatchMutation we don't need to
re-acquire it. The solution would be to keep a cache of the rowKeys already
locked for a miniBatchMutation and If we already have the
rowKey in the cache, we don't repeatedly try and acquire the lock. A fix to
this problem would be to keep a set of rows we already locked and not try to
acquire the lock for these rows.
We have tested this fix in our production environment and has improved
replication performance quite a bit. We saw a replication batch go from 3+
minutes to less than 10 seconds for batches with duplicate row keys.
{code}
static final int ACQUIRE_LOCK_COUNT = 0;
@Test
public void testRedundantRowKeys() throws Exception {
final int batchSize = 10;
String tableName = getClass().getSimpleName();
Configuration conf = HBaseConfiguration.create();
conf.setClass(HConstants.REGION_IMPL, MockHRegion.class, HeapSize.class);
MockHRegion region = (MockHRegion)
TestHRegion.initHRegion(Bytes.toBytes(tableName), tableName, conf,
Bytes.toBytes(a));
ListPairMutation, Integer someBatch = Lists.newArrayList();
int i = 0;
while (i batchSize) {
if (i % 2 == 0) {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(0)),
null));
} else {
someBatch.add(new PairMutation, Integer(new Put(Bytes.toBytes(1)),
null));
}
i++;
}
long startTime = System.currentTimeMillis();
region.batchMutate(someBatch.toArray(new Pair[0]));
long endTime = System.currentTimeMillis();
long duration = endTime - startTime;
System.out.println(duration: + duration + ms);
assertEquals(2, ACQUIRE_LOCK_COUNT);
}
@Override
public Integer getLock(Integer lockid, byte[] row, boolean waitForLock)
throws IOException {
ACQUIRE_LOCK_COUNT++;
return super.getLock(lockid, row, waitForLock);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8909) HBASE-6170 broke hbase.regionserver.lease.period config, we should support it for BC
[
https://issues.apache.org/jira/browse/HBASE-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708839#comment-13708839
]
Hadoop QA commented on HBASE-8909:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592365/hbase-8909_v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6344//console
This message is automatically generated.
HBASE-6170 broke hbase.regionserver.lease.period config, we should support
it for BC
--
Key: HBASE-8909
URL: https://issues.apache.org/jira/browse/HBASE-8909
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: hbase-8909_v1.patch, hbase-8909_v2.patch,
hbase-8909_v3.patch
HBASE-6170 replaces hbase.regionserver.lease.period with the new
configuration hbase.client.scanner.timeout.period. We should support the
old configuration at least in 0.96. We can put a WARN and deprecate the old
value. Remove it in 1.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8944) Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-8944:
--
Attachment: 8944.patch
What I committed to trunk. Adds a new section to troubleshooting section for
unit and integration tests. Eyeballed output html locally with Firefox.
Runtime NullPointerException in MiniDFSCluster on CentOS and other Linux
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
Assignee: Andrew Purtell
Attachments: 8944.patch, 8944.patch
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8944) Document workaround for MiniDFSCluster exceptions when running tests
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-8944.
---
Resolution: Fixed
Fix Version/s: 0.98.0
Document workaround for MiniDFSCluster exceptions when running tests
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
Assignee: Andrew Purtell
Fix For: 0.98.0
Attachments: 8944.patch, 8944.patch
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8944) Document workaround for MiniDFSCluster exceptions when running tests
[
https://issues.apache.org/jira/browse/HBASE-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-8944:
--
Summary: Document workaround for MiniDFSCluster exceptions when running
tests (was: Runtime NullPointerException in MiniDFSCluster on CentOS and other
Linux)
Document workaround for MiniDFSCluster exceptions when running tests
Key: HBASE-8944
URL: https://issues.apache.org/jira/browse/HBASE-8944
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.6, 0.94.6.1
Reporter: David Williams
Assignee: Andrew Purtell
Attachments: 8944.patch, 8944.patch
The HBaseTestingUtility appears to have an unhandled NullPointerException in
certain environments. It works on Apache HBase 0.94.6.1 on Mac OSX but not
in certain Linux environments such as CentOS.
In Java:
{code}
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster(1); // blows up on CentOS Linux but not Mac
OSX
{code}
Creates this exception:
{code}
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
(MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.init (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
(HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
(HBaseTestingUtility.java:555)
{code}
I also created a stack overflow ticket here, with more info:
http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments
Steps to reproduce:
1. Download a copy of CentOS
2. Install Java
3. Fire up a minicluster as part of the HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8670) [0.94] Backport HBASE-8449 and HBASE-8204 to 0.94 (Refactor recoverLease retries and pauses)
[ https://issues.apache.org/jira/browse/HBASE-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708854#comment-13708854 ] Himanshu Vashishtha commented on HBASE-8670: +1 for 0.94.10. I will work on a patch in case you are busy, [~enis]. Thanks. [0.94] Backport HBASE-8449 and HBASE-8204 to 0.94 (Refactor recoverLease retries and pauses) Key: HBASE-8670 URL: https://issues.apache.org/jira/browse/HBASE-8670 Project: HBase Issue Type: Bug Components: Filesystem Integration, master, wal Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.94.10 Attachments: hbase-8670_v1.patch Some history: Up until 0.94.8, Hbase did not check the result of recoverLease() call, but things kind of worked since we are checking for 0-length files in distributed log split tasks from region servers. If lease recovery is not finished, the log file will report 0 length, and the task will fail, and master will then re-call recoverLease() and reassign the task. This scheme might fail for log files that are larger than 1 hdfs block though. In 0.94.8, we committed (HBASE-8354, which is backport of HBASE-7878) and later increased the sleep time to 4 secs in HBASE-8389. However, the proper solution arrived in trunk in HBASE-8449 which uses a backoff sleep policy + isFileClosed() api. We should backport this patch to 0.94 as well. isFileClosed() is released in Hadoop 1.2.0 (HDFS-4774) and 2.0.5(HDFS-4525). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8877) Reentrant row locks
[
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708851#comment-13708851
]
Dave Latham commented on HBASE-8877:
[~lhofhansl] Agreed that it's a bit fragile and that exactly was why I was
initially concerned about checkAndMutate before I convinced myself it was safe
and added the comment indicating that the nested release may have already
released the locks.
Reference counts would change the contract (from current patch) and require
clients like doMiniBatchMutation to track the number of times it acquires the
lock on each row and then release them the same number of times as opposed to a
simple releaseAll. I can give it a shot to see how it turns out if you want to
see it.
Reentrant row locks
---
Key: HBASE-8877
URL: https://issues.apache.org/jira/browse/HBASE-8877
Project: HBase
Issue Type: Bug
Components: Coprocessors, regionserver
Reporter: Dave Latham
Assignee: Dave Latham
Fix For: 0.95.2
Attachments: hbase-8877-0.94-microbenchmark.txt,
HBASE-8877-0.94.patch, HBASE-8877-0.94-v2.patch, HBASE-8877.patch,
HBASE-8877-v2.patch, HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt,
HBASE-8877-v4.patch, HBASE-8877-v5.patch, HBASE-8877-v6.patch,
HBASE-8877-v7.patch
HBASE-8806 revealed performance problems with batch mutations failing to
reacquire the same row locks. It looks like HBASE-8806 will use a less
intrusive change for 0.94 to have batch mutations track their own row locks
and not attempt to reacquire them. Another approach will be to support
reentrant row locks directly. This allows simplifying a great deal of
calling code to no longer track and pass around lock ids.
One affect this change will have is changing the RegionObserver coprocessor's
methods preBatchMutate and postBatchMutate from taking a
{{MiniBatchOperationInProgressPairMutation, Integer miniBatchOp}} to
taking a {{MiniBatchOperationInProgressMutation miniBatchOp}}. I don't
believe CPs should be relying on these lock ids, but that's a potential
incompatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8691) High-Throughput Streaming Scan API
[ https://issues.apache.org/jira/browse/HBASE-8691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708855#comment-13708855 ] Sandy Pratt commented on HBASE-8691: Since my last comment, I've worked out some issues around integrating the streaming scan API with Hive, and I've pushed it out on an experimental basis to a production cluster for testing. End-to-end, in a table scan situation, the streaming scan API turns out to be about 45% faster than the RPC scan API (a full table scan of my dataset took 31 about minutes with the streaming API versus about 45 minutes with the RCP API). Some of the tweaking I had to do to get to that point: - Refactored the streaming client scanner to conform to the standard AbstractClientScanner API (I had used an event-driven approach previously, where clients registered an onMessage interface and hit go) - Created a TableInputFormat/RecordReader/etc. that leverages the new API - Profiled my custom SerDe to iron out some surprising hotpots As noted earlier, I looks like performance when streaming from the RS is highly dependent on keeping the pipe saturated. Too much latency in any particular spot will cause bubbles, which kills performance. As written, my SerDe was wasting too many cycles doing date formatting and initializing HashMaps (they seem to make a system call to srand). Once I ironed those issues out, I did a comparison between the streaming scan API and the RPC scan API, both using the newly optimized SerDe, which is where I found the 45% performance improvement. If it's true that latency is key to performance here, that delta might go up with more modern CPUs (I have Xeon 5450s currently) as the overhead of Hive and the SerDe decrease relative to network speed. High-Throughput Streaming Scan API -- Key: HBASE-8691 URL: https://issues.apache.org/jira/browse/HBASE-8691 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.95.0 Reporter: Sandy Pratt Labels: perfomance, scan Attachments: HRegionServlet.java, README.txt, RecordReceiver.java, ScannerTest.java, StreamHRegionServer.java, StreamReceiverDirect.java, StreamServletDirect.java I've done some working testing various ways to refactor and optimize Scans in HBase, and have found that performance can be dramatically increased by the addition of a streaming scan API. The attached code constitutes a proof of concept that shows performance increases of almost 4x in some workloads. I'd appreciate testing, replication, and comments. If the approach seems viable, I think such an API should be built into some future version of HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere
[ https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708861#comment-13708861 ] stack commented on HBASE-8927: -- bq. ...with the exception of column timestamp precision, for which a more complex and safer scheme can be made as suggested by some comments above, and measuring time taken for short (normally expected to take single seconds or less) operations. Seems like good enough reasons to me for doing nano time. I took a look at jdk8 and the story is no different there in its description of system.nanotime. It adds a java.time package with classes like Instant w/ its nanosecond resolution http://download.java.net/jdk8/docs/api/java/time/Instant.html but it doesn't look too amenable given its long of seconds and then nanos inside a second (apart from it being jdk8) Could do something like Lars suggests above w/ left shift filling in bottom few bytes w/ an incrementing number; it'd be a bit of a pain to implement but would be nice avoiding clashes. Use nano time instead of mili time everywhere - Key: HBASE-8927 URL: https://issues.apache.org/jira/browse/HBASE-8927 Project: HBase Issue Type: Bug Reporter: stack Attachments: 8927.txt Less collisions and we are paying the price of a long anyways so might as well fill it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8662) [rest] support impersonation
[ https://issues.apache.org/jira/browse/HBASE-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708863#comment-13708863 ] Hudson commented on HBASE-8662: --- SUCCESS: Integrated in HBase-TRUNK #4254 (See [https://builds.apache.org/job/HBase-TRUNK/4254/]) HBASE-8662 [rest] support impersonation - ADDENDUM (jxiang: rev 1503360) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java [rest] support impersonation Key: HBASE-8662 URL: https://issues.apache.org/jira/browse/HBASE-8662 Project: HBase Issue Type: Sub-task Components: REST, security Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.98.0 Attachments: method_doas.patch, secure_rest.patch, trunk-8662.patch, trunk-8662_v2.patch, trunk-8662_v3.patch, trunk-8662_v4.patch, trunk-8662_v5.patch, trunk-8662_v6.patch, trunk-8662_v7.1.patch, trunk-8662_v7.2.patch, trunk-8662_v7.3.patch, trunk-8662_v7.patch Currently, our client API uses a fixed user: the current user. It should accept a user passed in, if authenticated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8952) Missing error handling can cause RegionServer RPC thread to busy loop forever
[ https://issues.apache.org/jira/browse/HBASE-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Sigoure updated HBASE-8952: -- Attachment: HBASE-8952.patch Patch to fix the issue in the 0.95 branch. Missing error handling can cause RegionServer RPC thread to busy loop forever - Key: HBASE-8952 URL: https://issues.apache.org/jira/browse/HBASE-8952 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Benoit Sigoure Assignee: Benoit Sigoure Attachments: HBASE-8952.patch This bug seems to be present in all released versions of HBase, including the tip of the 0.94 and 0.95 branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8662) [rest] support impersonation
[ https://issues.apache.org/jira/browse/HBASE-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708865#comment-13708865 ] Hudson commented on HBASE-8662: --- FAILURE: Integrated in hbase-0.95 #325 (See [https://builds.apache.org/job/hbase-0.95/325/]) HBASE-8662 [rest] support impersonation (jxiang: rev 1503341) * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java * /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java * /hbase/branches/0.95/hbase-protocol/src/main/protobuf/RPC.proto * /hbase/branches/0.95/hbase-server/pom.xml * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/Constants.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServlet.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServletContainer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/filter/AuthFilter.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * /hbase/branches/0.95/pom.xml [rest] support impersonation Key: HBASE-8662 URL: https://issues.apache.org/jira/browse/HBASE-8662 Project: HBase Issue Type: Sub-task Components: REST, security Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.98.0 Attachments: method_doas.patch, secure_rest.patch, trunk-8662.patch, trunk-8662_v2.patch, trunk-8662_v3.patch, trunk-8662_v4.patch, trunk-8662_v5.patch, trunk-8662_v6.patch, trunk-8662_v7.1.patch, trunk-8662_v7.2.patch, trunk-8662_v7.3.patch, trunk-8662_v7.patch Currently, our client API uses a fixed user: the current user. It should accept a user passed in, if authenticated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8952) Missing error handling can cause RegionServer RPC thread to busy loop forever
Benoit Sigoure created HBASE-8952: - Summary: Missing error handling can cause RegionServer RPC thread to busy loop forever Key: HBASE-8952 URL: https://issues.apache.org/jira/browse/HBASE-8952 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Benoit Sigoure Assignee: Benoit Sigoure Attachments: HBASE-8952.patch This bug seems to be present in all released versions of HBase, including the tip of the 0.94 and 0.95 branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8952) Missing error handling can cause RegionServer RPC thread to busy loop forever
[
https://issues.apache.org/jira/browse/HBASE-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708874#comment-13708874
]
Benoit Sigoure commented on HBASE-8952:
---
I attached a patch that fixes the issue for me in 0.95 – the patch would need
to be ported to other branches as well as to the secure RPC implementation,
which is separate in 0.94 ({{ipc/SecureServer.java}}).
Missing error handling can cause RegionServer RPC thread to busy loop forever
-
Key: HBASE-8952
URL: https://issues.apache.org/jira/browse/HBASE-8952
Project: HBase
Issue Type: Bug
Components: IPC/RPC
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Attachments: HBASE-8952.patch
This bug seems to be present in all released versions of HBase, including the
tip of the 0.94 and 0.95 branches.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8952) Missing error handling can cause RegionServer RPC thread to busy loop forever
[ https://issues.apache.org/jira/browse/HBASE-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Sigoure updated HBASE-8952: -- Status: Patch Available (was: Open) Missing error handling can cause RegionServer RPC thread to busy loop forever - Key: HBASE-8952 URL: https://issues.apache.org/jira/browse/HBASE-8952 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Benoit Sigoure Assignee: Benoit Sigoure Attachments: HBASE-8952.patch This bug seems to be present in all released versions of HBase, including the tip of the 0.94 and 0.95 branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8877) Reentrant row locks
[
https://issues.apache.org/jira/browse/HBASE-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708869#comment-13708869
]
stack commented on HBASE-8877:
--
bq. ...what if in the future a method does its own locking...
You have a use case in mind [~lhofhansl]?
Reentrant row locks
---
Key: HBASE-8877
URL: https://issues.apache.org/jira/browse/HBASE-8877
Project: HBase
Issue Type: Bug
Components: Coprocessors, regionserver
Reporter: Dave Latham
Assignee: Dave Latham
Fix For: 0.95.2
Attachments: hbase-8877-0.94-microbenchmark.txt,
HBASE-8877-0.94.patch, HBASE-8877-0.94-v2.patch, HBASE-8877.patch,
HBASE-8877-v2.patch, HBASE-8877-v3.patch, hbase-8877-v4-microbenchmark.txt,
HBASE-8877-v4.patch, HBASE-8877-v5.patch, HBASE-8877-v6.patch,
HBASE-8877-v7.patch
HBASE-8806 revealed performance problems with batch mutations failing to
reacquire the same row locks. It looks like HBASE-8806 will use a less
intrusive change for 0.94 to have batch mutations track their own row locks
and not attempt to reacquire them. Another approach will be to support
reentrant row locks directly. This allows simplifying a great deal of
calling code to no longer track and pass around lock ids.
One affect this change will have is changing the RegionObserver coprocessor's
methods preBatchMutate and postBatchMutate from taking a
{{MiniBatchOperationInProgressPairMutation, Integer miniBatchOp}} to
taking a {{MiniBatchOperationInProgressMutation miniBatchOp}}. I don't
believe CPs should be relying on these lock ids, but that's a potential
incompatibility.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8952) Missing error handling can cause RegionServer RPC thread to busy loop forever
[ https://issues.apache.org/jira/browse/HBASE-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Sigoure updated HBASE-8952: -- Description: If the connection to the client is closed unexpectedly and at the wrong time, the code will attempt to keep reading from the socket in a busy loop. This bug seems to be present in all released versions of HBase, including the tip of the 0.94 and 0.95 branches, however I only ran into it while porting AsyncHBase to 0.95 was:This bug seems to be present in all released versions of HBase, including the tip of the 0.94 and 0.95 branches. Missing error handling can cause RegionServer RPC thread to busy loop forever - Key: HBASE-8952 URL: https://issues.apache.org/jira/browse/HBASE-8952 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Benoit Sigoure Assignee: Benoit Sigoure Attachments: HBASE-8952.patch If the connection to the client is closed unexpectedly and at the wrong time, the code will attempt to keep reading from the socket in a busy loop. This bug seems to be present in all released versions of HBase, including the tip of the 0.94 and 0.95 branches, however I only ran into it while porting AsyncHBase to 0.95 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
