[jira] [Comment Edited] (HBASE-17757) Unify blocksize after encoding to decrease memory fragment
[ https://issues.apache.org/jira/browse/HBASE-17757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900866#comment-15900866 ] Anoop Sam John edited comment on HBASE-17757 at 3/8/17 7:51 AM: In case where both DBE and compression in use, the size we track will be after compression also. And what we keep in cache is uncompressed blocks (By default. There is config to keep compressed also).. So the math will go wrong there? Now also the block sizes for BucketCache has to be tuned considering the encoding factor. Ya it makes the size restrictions bit more clear. So when there is compression also in place, need to adjust the factor accordingly say 0.5 to 0.3 (eg: ) was (Author: anoop.hbase): In case where both DBE and compression in use, the size we track will be after compression also. And what we keep in cache is uncompressed blocks (By default. There is config to keep compressed also).. So the math will go wrong there? Now also the block sizes for BucketCache has to be tuned considering the encoding factor. Ya it makes the size restrictions bit more clear. So when there is compression also in place, need to adjust the factor accordingly say 0.5 to 0.3 (eg:) > Unify blocksize after encoding to decrease memory fragment > --- > > Key: HBASE-17757 > URL: https://issues.apache.org/jira/browse/HBASE-17757 > Project: HBase > Issue Type: New Feature >Reporter: Allan Yang >Assignee: Allan Yang > Attachments: HBASE-17757.patch > > > Usually, we store encoded block(uncompressed) in blockcache/bucketCache. > Though we have set the blocksize, after encoding, blocksize is varied. Varied > blocksize will cause memory fragment problem, which will result in more FGC > finally.In order to relief the memory fragment, This issue adjusts the > encoded block to a unified size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17757) Unify blocksize after encoding to decrease memory fragment
[ https://issues.apache.org/jira/browse/HBASE-17757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900867#comment-15900867 ] Anoop Sam John commented on HBASE-17757: How abt thinking the block size limit to be a hard limit than a soft one? I mean dont allow to cross the limit at all. We have 1K size adjustment in BC bucket sizes to adjust this possible overgrow + header /checksum... (We remove this extra bytes like header etc now while caching to BC? Keeps forgetting this.. [~saint@gmail.com] ) > Unify blocksize after encoding to decrease memory fragment > --- > > Key: HBASE-17757 > URL: https://issues.apache.org/jira/browse/HBASE-17757 > Project: HBase > Issue Type: New Feature >Reporter: Allan Yang >Assignee: Allan Yang > Attachments: HBASE-17757.patch > > > Usually, we store encoded block(uncompressed) in blockcache/bucketCache. > Though we have set the blocksize, after encoding, blocksize is varied. Varied > blocksize will cause memory fragment problem, which will result in more FGC > finally.In order to relief the memory fragment, This issue adjusts the > encoded block to a unified size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17717) Incorrect ZK ACL set for HBase superuser
[
https://issues.apache.org/jira/browse/HBASE-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900865#comment-15900865
]
Mikhail Antonov commented on HBASE-17717:
-
Thanks [~elserj]! +1 for branch-1.3, sorry for a bit delayed reply.
> Incorrect ZK ACL set for HBase superuser
>
>
> Key: HBASE-17717
> URL: https://issues.apache.org/jira/browse/HBASE-17717
> Project: HBase
> Issue Type: Bug
> Components: security, Zookeeper
>Reporter: Shreya Bhat
>Assignee: Josh Elser
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.1.10, 1.2.6
>
> Attachments: HBASE-17717.001.0.98.patch,
> HBASE-17717.001.branch-1.1.patch, HBASE-17717.001.patch
>
>
> Shreya was doing some testing of a deploy of HBase, verifying that the ZK
> ACLs were actually set as we expect (yay, security).
> She noticed that, in some cases, we were seeing multiple ACLs for the same
> user.
> {noformat}
> 'world,'anyone
> : r
> 'sasl,'hbase
> : cdrwa
> 'sasl,'hbase
> : cdrwa
> {noformat}
> After digging into this (and some insight from the mighty [~enis]), we
> realized that this was happening because of an overridden value for
> {{hbase.superuser}}. However, the ACL value doesn't match what we'd expect to
> see (as hbase.superuser was set to {{cstm-hbase}}).
> After digging into this code, it seems like the {{auth}} ACL scheme in
> ZooKeeper does not work as we expect.
> {code}
> if (superUser != null) {
> acls.add(new ACL(Perms.ALL, new Id("auth", superUser)));
> }
> {code}
> In the above, the {{"auth"}} scheme ignores any provided "subject" in the
> {{Id}} object. It *only* considers the authentication of the current
> connection. As such, our usage of this never actually sets the ACL for the
> superuser correctly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17746) TestSimpleRpcScheduler.testCoDelScheduling is broken
[ https://issues.apache.org/jira/browse/HBASE-17746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CHIA-PING TSAI updated HBASE-17746: --- Fix Version/s: 1.4.0 2.0.0 > TestSimpleRpcScheduler.testCoDelScheduling is broken > > > Key: HBASE-17746 > URL: https://issues.apache.org/jira/browse/HBASE-17746 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0 > Environment: OS: Ubuntu 14.04 > Arch: x86_64 > Linux x 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 > x86_64 x86_64 x86_64 GNU/Linux >Reporter: Anup Halarnkar > Fix For: 2.0.0, 1.4.0 > > > Command: > export JAVA_TOOL_OPTIONS="-Xmx8G -Xms2G" > mvn clean install -X > Output: > Results : > Failed tests: > TestSimpleRpcScheduler.testCoDelScheduling:469 None of these calls should > have been discarded expected:<0> but was:<15> > Tests run: 1191, Failures: 1, Errors: 0, Skipped: 12 > Output of "TestSimpleRpcScheduler.txt" > --- > Test set: org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > --- > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.642 sec <<< > FAILURE! - in org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > testCoDelScheduling(org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler) Time > elapsed: 0.563 sec <<< FAILURE! > java.lang.AssertionError: None of these calls should have been discarded > expected:<0> but was:<15> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testCoDelScheduling(TestSimpleRpcScheduler.java:469) > -- > Output of last few lines of "TestSimpleRpcScheduler-output.txt" > 2017-03-07 16:09:16,580 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testCoDelScheduling Thread=5 (was 5), > OpenFileDescriptor=250 (was 250), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 157) - ProcessCount > LEAK? -, AvailableMemoryMB=1734 (was 1762) > 2017-03-07 16:09:16,604 INFO [main] hbase.ResourceChecker(148): before: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5, > OpenFileDescriptor=252, MaxFileDescriptor=4096, SystemLoadAverage=607, > ProcessCount=158, AvailableMemoryMB=1734 > 2017-03-07 16:09:16,722 INFO [main] ipc.RpcExecutor(145): RpcExecutor name > using fifo as call queue; numCallQueues=1; maxQueueLength=5; handlerCount=0 > 2017-03-07 16:09:16,722 DEBUG [main] ipc.RpcExecutor(215): Started > RpcServer.deafult.FPBQ.Fifo.handler=0,queue=0,port=0 > 2017-03-07 16:09:16,839 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5 (was 5), > OpenFileDescriptor=250 (was 252), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 158), > AvailableMemoryMB=1703 (was 1734) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17757) Unify blocksize after encoding to decrease memory fragment
[ https://issues.apache.org/jira/browse/HBASE-17757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900866#comment-15900866 ] Anoop Sam John commented on HBASE-17757: In case where both DBE and compression in use, the size we track will be after compression also. And what we keep in cache is uncompressed blocks (By default. There is config to keep compressed also).. So the math will go wrong there? Now also the block sizes for BucketCache has to be tuned considering the encoding factor. Ya it makes the size restrictions bit more clear. So when there is compression also in place, need to adjust the factor accordingly say 0.5 to 0.3 (eg:) > Unify blocksize after encoding to decrease memory fragment > --- > > Key: HBASE-17757 > URL: https://issues.apache.org/jira/browse/HBASE-17757 > Project: HBase > Issue Type: New Feature >Reporter: Allan Yang >Assignee: Allan Yang > Attachments: HBASE-17757.patch > > > Usually, we store encoded block(uncompressed) in blockcache/bucketCache. > Though we have set the blocksize, after encoding, blocksize is varied. Varied > blocksize will cause memory fragment problem, which will result in more FGC > finally.In order to relief the memory fragment, This issue adjusts the > encoded block to a unified size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17746) TestSimpleRpcScheduler.testCoDelScheduling is broken
[ https://issues.apache.org/jira/browse/HBASE-17746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CHIA-PING TSAI updated HBASE-17746: --- Summary: TestSimpleRpcScheduler.testCoDelScheduling is broken (was: TestSimpleRpcScheduler.testCoDelScheduling:469 None of these calls should have been discarded expected:<0> but was:<15>) > TestSimpleRpcScheduler.testCoDelScheduling is broken > > > Key: HBASE-17746 > URL: https://issues.apache.org/jira/browse/HBASE-17746 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0 > Environment: OS: Ubuntu 14.04 > Arch: x86_64 > Linux x 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 > x86_64 x86_64 x86_64 GNU/Linux >Reporter: Anup Halarnkar > Fix For: 2.0.0, 1.4.0 > > > Command: > export JAVA_TOOL_OPTIONS="-Xmx8G -Xms2G" > mvn clean install -X > Output: > Results : > Failed tests: > TestSimpleRpcScheduler.testCoDelScheduling:469 None of these calls should > have been discarded expected:<0> but was:<15> > Tests run: 1191, Failures: 1, Errors: 0, Skipped: 12 > Output of "TestSimpleRpcScheduler.txt" > --- > Test set: org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > --- > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.642 sec <<< > FAILURE! - in org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > testCoDelScheduling(org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler) Time > elapsed: 0.563 sec <<< FAILURE! > java.lang.AssertionError: None of these calls should have been discarded > expected:<0> but was:<15> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testCoDelScheduling(TestSimpleRpcScheduler.java:469) > -- > Output of last few lines of "TestSimpleRpcScheduler-output.txt" > 2017-03-07 16:09:16,580 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testCoDelScheduling Thread=5 (was 5), > OpenFileDescriptor=250 (was 250), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 157) - ProcessCount > LEAK? -, AvailableMemoryMB=1734 (was 1762) > 2017-03-07 16:09:16,604 INFO [main] hbase.ResourceChecker(148): before: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5, > OpenFileDescriptor=252, MaxFileDescriptor=4096, SystemLoadAverage=607, > ProcessCount=158, AvailableMemoryMB=1734 > 2017-03-07 16:09:16,722 INFO [main] ipc.RpcExecutor(145): RpcExecutor name > using fifo as call queue; numCallQueues=1; maxQueueLength=5; handlerCount=0 > 2017-03-07 16:09:16,722 DEBUG [main] ipc.RpcExecutor(215): Started > RpcServer.deafult.FPBQ.Fifo.handler=0,queue=0,port=0 > 2017-03-07 16:09:16,839 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5 (was 5), > OpenFileDescriptor=250 (was 252), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 158), > AvailableMemoryMB=1703 (was 1734) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
Guangxu Cheng created HBASE-17758:
-
Summary: [RSGROUP] Add shell command to move servers and tables at
the same time
Key: HBASE-17758
URL: https://issues.apache.org/jira/browse/HBASE-17758
Project: HBase
Issue Type: New Feature
Components: rsgroup
Reporter: Guangxu Cheng
Currently add a new group perform the following steps:
{code:javascript}
hbase(main):001:0> add_rsgroup 'mygroup'
Took 0.3840 seconds
hbase(main):002:0> move_servers_rsgroup
'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
Took 3.5040 seconds
hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
Took 0.2710 seconds
{code}
1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and hbase-rs-02
2. move_tables_rsgroup will also reassign all the regions of the table example.
This will lead to a large number of regions to migrate and affect the data
locality.
However, some regions of the table that are distributed on hbase-rs-01 and
hbase-rs-02, do not need to be moved.
So,we need a new shell command *move_servers_tables_rsgroup*.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17669) Implement async mergeRegion/splitRegion methods.
[ https://issues.apache.org/jira/browse/HBASE-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900846#comment-15900846 ] Duo Zhang commented on HBASE-17669: --- The UTs you added are failing? > Implement async mergeRegion/splitRegion methods. > > > Key: HBASE-17669 > URL: https://issues.apache.org/jira/browse/HBASE-17669 > Project: HBase > Issue Type: Sub-task > Components: Admin, asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0 > > Attachments: HBASE-17669.v1.patch > > > RT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16584: -- Attachment: HBASE-16584-branch-1-v1.patch Temporarily disable TestSecureIPC.testSaslNoCommonQop. The problem is in hadoop, where UGI has some problems working along with JDK8. > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch, HBASE-16584-branch-1-v1.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17757) Unify blocksize after encoding to decrease memory fragment
[ https://issues.apache.org/jira/browse/HBASE-17757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-17757: --- Attachment: HBASE-17757.patch > Unify blocksize after encoding to decrease memory fragment > --- > > Key: HBASE-17757 > URL: https://issues.apache.org/jira/browse/HBASE-17757 > Project: HBase > Issue Type: New Feature >Reporter: Allan Yang >Assignee: Allan Yang > Attachments: HBASE-17757.patch > > > Usually, we store encoded block(uncompressed) in blockcache/bucketCache. > Though we have set the blocksize, after encoding, blocksize is varied. Varied > blocksize will cause memory fragment problem, which will result in more FGC > finally.In order to relief the memory fragment, This issue adjusts the > encoded block to a unified size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900828#comment-15900828 ] Allan Yang commented on HBASE-17739: Here is the Issue https://issues.apache.org/jira/browse/HBASE-17757 > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17757) Unify blocksize after encoding to decrease memory fragment
[ https://issues.apache.org/jira/browse/HBASE-17757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-17757: --- Status: Patch Available (was: Open) > Unify blocksize after encoding to decrease memory fragment > --- > > Key: HBASE-17757 > URL: https://issues.apache.org/jira/browse/HBASE-17757 > Project: HBase > Issue Type: New Feature >Reporter: Allan Yang >Assignee: Allan Yang > Attachments: HBASE-17757.patch > > > Usually, we store encoded block(uncompressed) in blockcache/bucketCache. > Though we have set the blocksize, after encoding, blocksize is varied. Varied > blocksize will cause memory fragment problem, which will result in more FGC > finally.In order to relief the memory fragment, This issue adjusts the > encoded block to a unified size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17757) Unify blocksize after encoding to decrease memory fragment
Allan Yang created HBASE-17757: -- Summary: Unify blocksize after encoding to decrease memory fragment Key: HBASE-17757 URL: https://issues.apache.org/jira/browse/HBASE-17757 Project: HBase Issue Type: New Feature Reporter: Allan Yang Assignee: Allan Yang Usually, we store encoded block(uncompressed) in blockcache/bucketCache. Though we have set the blocksize, after encoding, blocksize is varied. Varied blocksize will cause memory fragment problem, which will result in more FGC finally.In order to relief the memory fragment, This issue adjusts the encoded block to a unified size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17712) Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound
[ https://issues.apache.org/jira/browse/HBASE-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17712: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-1. Thanks [~stack] for reviewing. > Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound > - > > Key: HBASE-17712 > URL: https://issues.apache.org/jira/browse/HBASE-17712 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17712-branch-1.patch, HBASE-17712.patch, > HBASE-17712-ut.patch, HBASE-17712-v1.patch, HBASE-17712-v2.patch, > HBASE-17712-v3.patch > > > It is introduced in HBASE-13651 and the logic became much more complicated > after HBASE-16304 due to a dead lock issue. It is really tough as sequence id > is involved in and the method we called is used to serve secondary replica > originally which does not handle write. > In fact, in 1.x release, the problem described in HBASE-13651 is gone. Now we > will write a compaction marker to WAL before deleting the compacted files. We > can only consider a RS as dead after its WAL files are all closed so if the > region has already been reassigned the compaction will fail as we can not > write out the compaction marker. > So theoretically, if we still hit FileNotFound exception, it should be a > critical bug which means we may loss data. I do not think it is a good idea > to just eat the exception and refresh store files. Or even if we want to do > this, we can just refresh store files without dropping memstore contents. > This will also simplify the logic a lot. > Suggestions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[ https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900813#comment-15900813 ] Yechao Chen commented on HBASE-17737: - [~tedyu] Yes,1 is better than 5.thanks for review. update the patch ,lines 841,847 use handler.getScannerRows(scanId, 1) instead of handler.getScannerRows(scanId, 5) > Thrift2 proxy should support scan timeRange per column family > - > > Key: HBASE-17737 > URL: https://issues.apache.org/jira/browse/HBASE-17737 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17737.patch, HBASE-17737-v1.patch, > HBASE-17737-v2.patch > > > see discussion in HBASE-14872 > and HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[ https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HBASE-17737: Attachment: HBASE-17737-v2.patch > Thrift2 proxy should support scan timeRange per column family > - > > Key: HBASE-17737 > URL: https://issues.apache.org/jira/browse/HBASE-17737 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17737.patch, HBASE-17737-v1.patch, > HBASE-17737-v2.patch > > > see discussion in HBASE-14872 > and HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[ https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900815#comment-15900815 ] Anoop Sam John commented on HBASE-17747: I see.. Missed that change as it was in btw 2 test files and I thought code wise change is over before that :-) So in normal mode (not embedded) how the perf of read from BC looks like? U see any change? > Support both weak and soft object pool > -- > > Key: HBASE-17747 > URL: https://issues.apache.org/jira/browse/HBASE-17747 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17747.patch, HBASE-17747.v2.patch > > > During YCSB testing on embedded mode after HBASE-17744, we found that under > high read load GC is quite severe even with offheap L2 cache. After some > investigation, we found it's caused by using weak reference for > IdReadWriteLock. In embedded mode the read is so quick that the lock might > already get promoted to the old generation when the weak reference is > cleared, which causes dirty card table thus slowing YGC. > So we proposed to use soft reference for this IdReadWriteLock used in cache, > which won't get cleared until JVM memory is not enough, and could resolve the > issue mentioned above. What's more, we propose to extend the WeakObjectPool > to be more generate to support both weak and soft reference. > Note that this issue only emerges under embedded mode with DirectOperator, in > which case all costs on the wire is removed thus produces extremely high > workloads. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[
https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900811#comment-15900811
]
Hadoop QA commented on HBASE-17584:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m 23s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 31s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 1s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856738/HBASE-17584-v1.patch |
| JIRA Issue | HBASE-17584 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 99b6f18a8f09 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c42a066 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5999/testReport/ |
| modules | C: hbase-client hbase-server hbase-rest U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5999/console |
| Powered
[jira] [Commented] (HBASE-17600) Implement get/create/modify/delete/list namespace admin operations
[
https://issues.apache.org/jira/browse/HBASE-17600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900807#comment-15900807
]
Hadoop QA commented on HBASE-17600:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 36s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 104m 0s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856736/HBASE-17600.master.004.patch
|
| JIRA Issue | HBASE-17600 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6414a95a1691 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c42a066 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5998/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5998/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Implement get/create/modify/delete/list
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[
https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900787#comment-15900787
]
Hadoop QA commented on HBASE-17747:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color}
| {color:red} HBASE-17747 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856752/HBASE-17747.v2.patch |
| JIRA Issue | HBASE-17747 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6000/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Support both weak and soft object pool
> --
>
> Key: HBASE-17747
> URL: https://issues.apache.org/jira/browse/HBASE-17747
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17747.patch, HBASE-17747.v2.patch
>
>
> During YCSB testing on embedded mode after HBASE-17744, we found that under
> high read load GC is quite severe even with offheap L2 cache. After some
> investigation, we found it's caused by using weak reference for
> IdReadWriteLock. In embedded mode the read is so quick that the lock might
> already get promoted to the old generation when the weak reference is
> cleared, which causes dirty card table thus slowing YGC.
> So we proposed to use soft reference for this IdReadWriteLock used in cache,
> which won't get cleared until JVM memory is not enough, and could resolve the
> issue mentioned above. What's more, we propose to extend the WeakObjectPool
> to be more generate to support both weak and soft reference.
> Note that this issue only emerges under embedded mode with DirectOperator, in
> which case all costs on the wire is removed thus produces extremely high
> workloads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17712) Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound
[ https://issues.apache.org/jira/browse/HBASE-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900785#comment-15900785 ] Duo Zhang commented on HBASE-17712: --- The failed UTs are unrelated and can pass locally. Will commit shortly. > Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound > - > > Key: HBASE-17712 > URL: https://issues.apache.org/jira/browse/HBASE-17712 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17712-branch-1.patch, HBASE-17712.patch, > HBASE-17712-ut.patch, HBASE-17712-v1.patch, HBASE-17712-v2.patch, > HBASE-17712-v3.patch > > > It is introduced in HBASE-13651 and the logic became much more complicated > after HBASE-16304 due to a dead lock issue. It is really tough as sequence id > is involved in and the method we called is used to serve secondary replica > originally which does not handle write. > In fact, in 1.x release, the problem described in HBASE-13651 is gone. Now we > will write a compaction marker to WAL before deleting the compacted files. We > can only consider a RS as dead after its WAL files are all closed so if the > region has already been reassigned the compaction will fail as we can not > write out the compaction marker. > So theoretically, if we still hit FileNotFound exception, it should be a > critical bug which means we may loss data. I do not think it is a good idea > to just eat the exception and refresh store files. Or even if we want to do > this, we can just refresh store files without dropping memstore contents. > This will also simplify the logic a lot. > Suggestions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17747) Support both weak and soft object pool
[ https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17747: -- Attachment: HBASE-17747.v2.patch The v2 patch fixes the javadoc issue > Support both weak and soft object pool > -- > > Key: HBASE-17747 > URL: https://issues.apache.org/jira/browse/HBASE-17747 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17747.patch, HBASE-17747.v2.patch > > > During YCSB testing on embedded mode after HBASE-17744, we found that under > high read load GC is quite severe even with offheap L2 cache. After some > investigation, we found it's caused by using weak reference for > IdReadWriteLock. In embedded mode the read is so quick that the lock might > already get promoted to the old generation when the weak reference is > cleared, which causes dirty card table thus slowing YGC. > So we proposed to use soft reference for this IdReadWriteLock used in cache, > which won't get cleared until JVM memory is not enough, and could resolve the > issue mentioned above. What's more, we propose to extend the WeakObjectPool > to be more generate to support both weak and soft reference. > Note that this issue only emerges under embedded mode with DirectOperator, in > which case all costs on the wire is removed thus produces extremely high > workloads. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17712) Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound
[
https://issues.apache.org/jira/browse/HBASE-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900771#comment-15900771
]
Hadoop QA commented on HBASE-17712:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 57s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 24s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 20s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 117m 0s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapred.TestTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856737/HBASE-17712-branch-1.patch
|
| JIRA Issue | HBASE-17712 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 806196208b4e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 /
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[
https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900770#comment-15900770
]
Yu Li commented on HBASE-17747:
---
bq. So here there is no change in usage of soft ref instead of weak.. Just you
doing the abstraction and giving both impls so as to make a way for the
embedded mode usage.
{{IdReadWriteLock}} has been changed to use soft reference, please check the
change there and let me know if any concern sir [~anoop.hbase], thanks.
> Support both weak and soft object pool
> --
>
> Key: HBASE-17747
> URL: https://issues.apache.org/jira/browse/HBASE-17747
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17747.patch
>
>
> During YCSB testing on embedded mode after HBASE-17744, we found that under
> high read load GC is quite severe even with offheap L2 cache. After some
> investigation, we found it's caused by using weak reference for
> IdReadWriteLock. In embedded mode the read is so quick that the lock might
> already get promoted to the old generation when the weak reference is
> cleared, which causes dirty card table thus slowing YGC.
> So we proposed to use soft reference for this IdReadWriteLock used in cache,
> which won't get cleared until JVM memory is not enough, and could resolve the
> issue mentioned above. What's more, we propose to extend the WeakObjectPool
> to be more generate to support both weak and soft reference.
> Note that this issue only emerges under embedded mode with DirectOperator, in
> which case all costs on the wire is removed thus produces extremely high
> workloads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900762#comment-15900762
]
Anoop Sam John commented on HBASE-17623:
Ya pls.. Report back once u have the results.
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.branch-1.v1.patch, HBASE-17623.branch-1.v2.patch,
> HBASE-17623.branch-1.v2.patch, HBASE-17623.v0.patch, HBASE-17623.v1.patch,
> HBASE-17623.v1.patch, HBASE-17623.v2.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The onDiskBlockBytesWithHeader should maintain a bytes array which can be
> reused when building the hfile.
> # The onDiskBlockBytesWithHeader is copied to an new bytes array only when we
> need to cache the block.
> # If no block need to be cached, the uncompressedBlockBytesWithHeader will
> never be created.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[ https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900759#comment-15900759 ] Anoop Sam John commented on HBASE-17747: So here there is no change in usage of soft ref instead of weak.. Just you doing the abstraction and giving both impls so as to make a way for the embedded mode usage. Fine. +1 > Support both weak and soft object pool > -- > > Key: HBASE-17747 > URL: https://issues.apache.org/jira/browse/HBASE-17747 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17747.patch > > > During YCSB testing on embedded mode after HBASE-17744, we found that under > high read load GC is quite severe even with offheap L2 cache. After some > investigation, we found it's caused by using weak reference for > IdReadWriteLock. In embedded mode the read is so quick that the lock might > already get promoted to the old generation when the weak reference is > cleared, which causes dirty card table thus slowing YGC. > So we proposed to use soft reference for this IdReadWriteLock used in cache, > which won't get cleared until JVM memory is not enough, and could resolve the > issue mentioned above. What's more, we propose to extend the WeakObjectPool > to be more generate to support both weak and soft reference. > Note that this issue only emerges under embedded mode with DirectOperator, in > which case all costs on the wire is removed thus produces extremely high > workloads. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17623) Reuse the bytes array when building the hfile block
[
https://issues.apache.org/jira/browse/HBASE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900750#comment-15900750
]
CHIA-PING TSAI commented on HBASE-17623:
I'm running more tests without compaction. It will yield more information about
the connection between GC and runtime.
> Reuse the bytes array when building the hfile block
> ---
>
> Key: HBASE-17623
> URL: https://issues.apache.org/jira/browse/HBASE-17623
> Project: HBase
> Issue Type: Improvement
>Reporter: CHIA-PING TSAI
>Assignee: CHIA-PING TSAI
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: after(snappy_hfilesize=5.04GB).png,
> after(snappy_hfilesize=755MB).png, before(snappy_hfilesize=5.04GB).png,
> before(snappy_hfilesize=755MB).png, HBASE-17623.branch-1.v0.patch,
> HBASE-17623.branch-1.v1.patch, HBASE-17623.branch-1.v2.patch,
> HBASE-17623.branch-1.v2.patch, HBASE-17623.v0.patch, HBASE-17623.v1.patch,
> HBASE-17623.v1.patch, HBASE-17623.v2.patch, memory allocation measurement.xlsx
>
>
> There are two improvements.
> # The onDiskBlockBytesWithHeader should maintain a bytes array which can be
> reused when building the hfile.
> # The onDiskBlockBytesWithHeader is copied to an new bytes array only when we
> need to cache the block.
> # If no block need to be cached, the uncompressedBlockBytesWithHeader will
> never be created.
> {code:title=HFileBlock.java|borderStyle=solid}
> private void finishBlock() throws IOException {
> if (blockType == BlockType.DATA) {
> this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx,
> userDataStream,
> baosInMemory.getBuffer(), blockType);
> blockType = dataBlockEncodingCtx.getBlockType();
> }
> userDataStream.flush();
> // This does an array copy, so it is safe to cache this byte array when
> cache-on-write.
> // Header is still the empty, 'dummy' header that is yet to be filled
> out.
> uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
> prevOffset = prevOffsetByType[blockType.getId()];
> // We need to set state before we can package the block up for
> cache-on-write. In a way, the
> // block is ready, but not yet encoded or compressed.
> state = State.BLOCK_READY;
> if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA)
> {
> onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> } else {
> onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
> compressAndEncrypt(uncompressedBlockBytesWithHeader);
> }
> // Calculate how many bytes we need for checksum on the tail of the
> block.
> int numBytes = (int) ChecksumUtil.numBytes(
> onDiskBlockBytesWithHeader.length,
> fileContext.getBytesPerChecksum());
> // Put the header for the on disk bytes; header currently is
> unfilled-out
> putHeader(onDiskBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> // Set the header for the uncompressed bytes (for cache-on-write) --
> IFF different from
> // onDiskBlockBytesWithHeader array.
> if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
> putHeader(uncompressedBlockBytesWithHeader, 0,
> onDiskBlockBytesWithHeader.length + numBytes,
> uncompressedBlockBytesWithHeader.length,
> onDiskBlockBytesWithHeader.length);
> }
> if (onDiskChecksum.length != numBytes) {
> onDiskChecksum = new byte[numBytes];
> }
> ChecksumUtil.generateChecksums(
> onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
> onDiskChecksum, 0, fileContext.getChecksumType(),
> fileContext.getBytesPerChecksum());
> }{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900747#comment-15900747 ] Anoop Sam John commented on HBASE-17738: 100 GB of off heap BC means we will create 25600 DBBs each of size 4 MB. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900742#comment-15900742 ] ramkrishna.s.vasudevan commented on HBASE-17738: I tried creating around 300G of ByteBuffers and it takes more than 1 min to do it. I think we can speed it up. But with FileCache it does not happen so I believe as it is only about creating a file. Will check that to confirm once. > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900715#comment-15900715
]
Hadoop QA commented on HBASE-16584:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 35 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
14s {color} | {color:green} branch-1 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 14s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 32s {color}
| {color:red} hbase-server-jdk1.7.0_80 with JDK v1.7.0_80 generated 1 new + 4
unchanged - 1 fixed = 5 total (was 5) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 12s {color}
| {color:red} root-jdk1.7.0_80 with JDK v1.7.0_80 generated 1 new + 14
unchanged - 1 fixed = 15 total (was 15) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 13s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s
{color} | {color:red} hbase-client-jdk1.8.0_121 with JDK v1.8.0_121 generated 5
new + 13 unchanged - 0 fixed = 18 total (was 13) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Commented] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[
https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900677#comment-15900677
]
Duo Zhang commented on HBASE-17584:
---
OK, I found this in our documentation.
{noformat}
New APIs introduced in a patch version will only be added in a source
compatible way [1]: i.e. code that implements public APIs will continue to
compile.
{noformat}
So it seems that for a minor release we are allowed to add new APIs that breaks
source compatibility? Good.
> Expose ScanMetrics with ResultScanner rather than Scan
> --
>
> Key: HBASE-17584
> URL: https://issues.apache.org/jira/browse/HBASE-17584
> Project: HBase
> Issue Type: Sub-task
> Components: Client, mapreduce, scan
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17584.patch, HBASE-17584-v1.patch
>
>
> I think this have been discussed many times... It is a bad practice to
> directly modify the Scan object passed in when calling getScanner. The reason
> that we can not use a copy is we need to use the Scan object to expose scan
> metrics. So we need to find another way to expose the metrics.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17584) Expose ScanMetrics with ResultScanner rather than Scan
[ https://issues.apache.org/jira/browse/HBASE-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17584: -- Attachment: HBASE-17584-v1.patch Rebase. > Expose ScanMetrics with ResultScanner rather than Scan > -- > > Key: HBASE-17584 > URL: https://issues.apache.org/jira/browse/HBASE-17584 > Project: HBase > Issue Type: Sub-task > Components: Client, mapreduce, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17584.patch, HBASE-17584-v1.patch > > > I think this have been discussed many times... It is a bad practice to > directly modify the Scan object passed in when calling getScanner. The reason > that we can not use a copy is we need to use the Scan object to expose scan > metrics. So we need to find another way to expose the metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17712) Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound
[ https://issues.apache.org/jira/browse/HBASE-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17712: -- Attachment: HBASE-17712-branch-1.patch Patch for branch-1. > Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound > - > > Key: HBASE-17712 > URL: https://issues.apache.org/jira/browse/HBASE-17712 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17712-branch-1.patch, HBASE-17712.patch, > HBASE-17712-ut.patch, HBASE-17712-v1.patch, HBASE-17712-v2.patch, > HBASE-17712-v3.patch > > > It is introduced in HBASE-13651 and the logic became much more complicated > after HBASE-16304 due to a dead lock issue. It is really tough as sequence id > is involved in and the method we called is used to serve secondary replica > originally which does not handle write. > In fact, in 1.x release, the problem described in HBASE-13651 is gone. Now we > will write a compaction marker to WAL before deleting the compacted files. We > can only consider a RS as dead after its WAL files are all closed so if the > region has already been reassigned the compaction will fail as we can not > write out the compaction marker. > So theoretically, if we still hit FileNotFound exception, it should be a > critical bug which means we may loss data. I do not think it is a good idea > to just eat the exception and refresh store files. Or even if we want to do > this, we can just refresh store files without dropping memstore contents. > This will also simplify the logic a lot. > Suggestions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17600) Implement get/create/modify/delete/list namespace admin operations
[ https://issues.apache.org/jira/browse/HBASE-17600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17600: --- Attachment: HBASE-17600.master.004.patch > Implement get/create/modify/delete/list namespace admin operations > -- > > Key: HBASE-17600 > URL: https://issues.apache.org/jira/browse/HBASE-17600 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17600.master.001.patch, > HBASE-17600.master.002.patch, HBASE-17600.master.003.patch, > HBASE-17600.master.003.patch, HBASE-17600.master.004.patch, > HBASE-17600.master.004.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[
https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900654#comment-15900654
]
Hadoop QA commented on HBASE-17737:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 31s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856725/HBASE-17737-v1.patch |
| JIRA Issue | HBASE-17737 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e9b4e74bb8e2 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ee1549c |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5995/testReport/ |
| modules | C: hbase-thrift U: hbase-thrift |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5995/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Thrift2 proxy should support scan timeRange per column family
> -
>
> Key: HBASE-17737
> URL: https://issues.apache.org/jira/browse/HBASE-17737
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Yechao Chen
>Assignee: Yechao Chen
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17737.patch,
[jira] [Commented] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[
https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900628#comment-15900628
]
Ted Yu commented on HBASE-17737:
nit:
{code}
847 handler.getScannerRows(scanId, 5);
{code}
1 can be used in place of 5, right ?
Looks good otherwise, pending QA.
> Thrift2 proxy should support scan timeRange per column family
> -
>
> Key: HBASE-17737
> URL: https://issues.apache.org/jira/browse/HBASE-17737
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Yechao Chen
>Assignee: Yechao Chen
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17737.patch, HBASE-17737-v1.patch
>
>
> see discussion in HBASE-14872
> and HBASE-14355
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16584: -- Attachment: HBASE-16584-branch-1.patch Retry. > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16584: -- Attachment: (was: HBASE-16584-branch-1.patch) > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[ https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900617#comment-15900617 ] Yechao Chen edited comment on HBASE-17737 at 3/8/17 3:16 AM: - Wrap long line，add a test and fill out release note. Replaced explicit type with diamond operator. [~tedyu],thanks for reply and review. was (Author: chenyechao): Wrap long line，add a test > Thrift2 proxy should support scan timeRange per column family > - > > Key: HBASE-17737 > URL: https://issues.apache.org/jira/browse/HBASE-17737 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17737.patch, HBASE-17737-v1.patch > > > see discussion in HBASE-14872 > and HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[ https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HBASE-17737: Status: Patch Available (was: Open) > Thrift2 proxy should support scan timeRange per column family > - > > Key: HBASE-17737 > URL: https://issues.apache.org/jira/browse/HBASE-17737 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17737.patch, HBASE-17737-v1.patch > > > see discussion in HBASE-14872 > and HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[ https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HBASE-17737: Release Note: Thrift2 proxy support scan timeRange per column family > Thrift2 proxy should support scan timeRange per column family > - > > Key: HBASE-17737 > URL: https://issues.apache.org/jira/browse/HBASE-17737 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17737.patch, HBASE-17737-v1.patch > > > see discussion in HBASE-14872 > and HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[ https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HBASE-17737: Release Note: Thrift2 proxy supports scan timeRange per column family (was: Thrift2 proxy support scan timeRange per column family) > Thrift2 proxy should support scan timeRange per column family > - > > Key: HBASE-17737 > URL: https://issues.apache.org/jira/browse/HBASE-17737 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17737.patch, HBASE-17737-v1.patch > > > see discussion in HBASE-14872 > and HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[ https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yechao Chen updated HBASE-17737: Attachment: HBASE-17737-v1.patch Wrap long line，add a test > Thrift2 proxy should support scan timeRange per column family > - > > Key: HBASE-17737 > URL: https://issues.apache.org/jira/browse/HBASE-17737 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17737.patch, HBASE-17737-v1.patch > > > see discussion in HBASE-14872 > and HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17728) Create separate build target for Configuration classes
[
https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900606#comment-15900606
]
Ted Yu commented on HBASE-17728:
In hbase-configuration-loader.h :
{code}
const std::string kHBaseDefaultXml = "hbase-default.xml";
const std::string kHBaseSiteXml = "hbase-site.xml";
const std::string kHBaseDefauktConfPath = "/etc/hbase/conf";
{code}
It seems the declaration of string constants in the code base isn't consistent.
On initial survey, I didn't find the clear advantage of using constexpr. If you
can explain some more, that would be nice.
> Create separate build target for Configuration classes
> --
>
> Key: HBASE-17728
> URL: https://issues.apache.org/jira/browse/HBASE-17728
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt,
> 17728.v5.txt, 17728.v6.txt, 17728.v7.txt, 17728.v8.txt, 17728.v9.txt
>
>
> User is in security module.
> When User::isSecurityEnabled() is added, we need to query Configuration for
> security setting.
> However, this introduces a circular build dependency:
> BUILD FAILED: Cycle found: //connection:connection -> //security:security ->
> //core:core -> //connection:connection
> This issue is to create separate build target for Configuration which is
> depended upon by both core and security modules.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17718) Difference between RS's servername and its ephemeral node cause SSH stop working
[
https://issues.apache.org/jira/browse/HBASE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900598#comment-15900598
]
Hadoop QA commented on HBASE-17718:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 26s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 38s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 113m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856710/HBASE-17718.branch-1.002.patch
|
| JIRA Issue | HBASE-17718 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux afa629beddc0 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / 5f63093 |
| Default Java | 1.7.0_80 |
| Multi-JDK versions |
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900589#comment-15900589 ] Allan Yang commented on HBASE-17739: Right away, Sir! [~zjushch] [~stack] > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900583#comment-15900583 ] chunhui shen commented on HBASE-17739: -- It's a feature completed by my teammate [~allan163], he will open a new issue to talk about this. Thanks, sir > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Attachment: 17728.v9.txt > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt, > 17728.v5.txt, 17728.v6.txt, 17728.v7.txt, 17728.v8.txt, 17728.v9.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to create separate build target for Configuration which is > depended upon by both core and security modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900546#comment-15900546 ] Ted Yu commented on HBASE-17728: Patch v9 should be ready to go. > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt, > 17728.v5.txt, 17728.v6.txt, 17728.v7.txt, 17728.v8.txt, 17728.v9.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to create separate build target for Configuration which is > depended upon by both core and security modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17728) Create separate build target for Configuration classes
[
https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900526#comment-15900526
]
Enis Soztutar commented on HBASE-17728:
---
You should not use {{hbase::optional}} for passing the Configuration. You can
declare the method like this:
{code}
+ static bool isSecurityEnabled(const Configuration ) {
+return KERBEROS.compare(conf.Get("hbase.security.authentication", "")) ==
0;
+ }
{code}
Please go over EVERY review comment before putting up more patches. There is no
use in putting iterations of half-finished patches addressing only some
reviews. This is still not-addressed:
{quote}
Also the way string constants are declared and named are slightly different.
Check client.h for examples:
{quote}
> Create separate build target for Configuration classes
> --
>
> Key: HBASE-17728
> URL: https://issues.apache.org/jira/browse/HBASE-17728
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt,
> 17728.v5.txt, 17728.v6.txt, 17728.v7.txt, 17728.v8.txt
>
>
> User is in security module.
> When User::isSecurityEnabled() is added, we need to query Configuration for
> security setting.
> However, this introduces a circular build dependency:
> BUILD FAILED: Cycle found: //connection:connection -> //security:security ->
> //core:core -> //connection:connection
> This issue is to create separate build target for Configuration which is
> depended upon by both core and security modules.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Attachment: Backup-restoreinHBase2.0 (4).pdf > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: HBASE-7912 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v20.txt, 14123-master.v21.txt, > 14123-master.v24.txt, 14123-master.v25.txt, 14123-master.v27.txt, > 14123-master.v28.txt, 14123-master.v29.full.txt, 14123-master.v2.txt, > 14123-master.v30.txt, 14123-master.v31.txt, 14123-master.v32.txt, > 14123-master.v33.txt, 14123-master.v34.txt, 14123-master.v35.txt, > 14123-master.v36.txt, 14123-master.v37.txt, 14123-master.v38.txt, > 14123.master.v39.patch, 14123-master.v3.txt, 14123.master.v40.patch, > 14123.master.v41.patch, 14123.master.v42.patch, 14123.master.v44.patch, > 14123.master.v45.patch, 14123.master.v46.patch, 14123.master.v48.patch, > 14123.master.v49.patch, 14123.master.v50.patch, 14123.master.v51.patch, > 14123.master.v52.patch, 14123.master.v54.patch, 14123.master.v56.patch, > 14123.master.v57.patch, 14123.master.v58.patch, 14123.master.v59.patch, > 14123-master.v5.txt, 14123.master.v60.patch, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > Backup-restoreinHBase2.0 (1).pdf, Backup-restoreinHBase2.0 (3).pdf, > Backup-restoreinHBase2.0 (4).pdf, Backup-restoreinHBase2.0.pdf, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v10.patch, HBASE-14123-v11.patch, HBASE-14123-v12.patch, > HBASE-14123-v13.patch, HBASE-14123-v15.patch, HBASE-14123-v16.patch, > HBASE-14123-v1.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Release Note: Vladimir Rodionov, Feb 2017 *** Summary of work HBASE-14123 The new feature introduces new command-line extensions to the hbase command and, from the client side, is accessible through command-line only Operations: * Create full backup on a list of tables or backup set * Create incremental backup image for table list or backup set * Restore list of tables from a given backup image * Show current backup progress * Delete backup image and all related images * Show history of backups * Backup set operations: create backup set, add/remove table to/from backup set, etc In the current implementation, the feature is already usable, meaning that users can backup tables and restore them using provided command-line tools. Both: full and incremental backups are supported. The full list of JIRAs included in this mega patch can be found in three umbrella JIRAs: HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all resolved ones) *** What are the remaining work items All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414. They are split into 3 groups: BLOCKER, CRITICAL, MAJOR Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release. * BLOCKER * HBASE-14417 Incremental backup and bulk loading ( Patch available) * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables (Patch available) * HBASE-17133 Backup documentation * HBASE-15227 Fault tolerance support * CRITICAL * HBASE-16465 Disable split/merges during backup We have umbrella JIRA (HBASE-14414) to track all the remaining work All the BLOCKER and CRITICAL JIRAs currently in open state will be implemented by 2.0 release time. Some MAJOR too, but it depends on resource availability The former development branch (HBASE-7912) is obsolete and will be closed/deleted after the merge. We want backup to be a GA feature in 2.0 We are going to support full backward compatibility for backup tool in 2.0 and onwards. Configuration Backup is disabled, by default. To enable it, the following configuration properties must be added to hbase-site.xml: hbase.backup.enable=true hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.apache.hadoop.hbase.backup.master.BackupLogCleaner hbase.procedure.master.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager hbase.procedure.regionserver.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager Known Issues * Bulk loaded data is not visible for incremental backup (will be resolved by 2.0 release) * After first backup sessions, WAL files will be accumulated in old WAL directory until the next backup sessions. Make sure, that you either run backups regularly or disable backups in the system. * No fault tolerance yet, some backup failures may result in corrupted backup data and will require manual intervention (will be resolved by 2.0 release) was: Vladimir Rodionov, Feb 2017 *** Summary of work The new feature introduces new command-line extensions to the hbase command and, from the client side, is accessible through command-line only Operations: * Create full backup on a list of tables or backup set * Create incremental backup image for table list or backup set * Restore list of tables from a given backup image * Show current backup progress * Delete backup image and all related images * Show history of backups * Backup set operations: create backup set, add/remove table to/from backup set, etc In the current implementation, the feature is already usable, meaning that users can backup tables and restore them using provided command-line tools. Both: full and incremental backups are supported. The full list of JIRAs included in this mega patch can be found in three umbrella JIRAs: HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all resolved ones) *** What are the remaining work items All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414. They are split into 3 groups: BLOCKER, CRITICAL, MAJOR Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release. * BLOCKER * HBASE-14417 Incremental backup and bulk loading ( Patch available) * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables (Patch available) * HBASE-17133 Backup documentation * HBASE-15227 Fault tolerance support * CRITICAL * HBASE-16465 Disable split/merges during backup We have umbrella JIRA (HBASE-14414) to track all the remaining work All the BLOCKER and CRITICAL JIRAs currently in
[jira] [Updated] (HBASE-17718) Difference between RS's servername and its ephemeral node cause SSH stop working
[
https://issues.apache.org/jira/browse/HBASE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17718:
--
Attachment: HBASE-17718.branch-1.002.patch
> Difference between RS's servername and its ephemeral node cause SSH stop
> working
>
>
> Key: HBASE-17718
> URL: https://issues.apache.org/jira/browse/HBASE-17718
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.4, 1.1.8
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17718.branch-1.001.patch,
> HBASE-17718.branch-1.002.patch, HBASE-17718.master.001.patch,
> HBASE-17718.master.002.patch, HBASE-17718.master.003.patch
>
>
> After HBASE-9593, RS put up an ephemeral node in ZK before reporting for
> duty. But if the hosts config (/etc/hosts) is different between master and
> RS, RS's serverName can be different from the one stored the ephemeral zk
> node. The email metioned in HBASE-13753
> (http://mail-archives.apache.org/mod_mbox/hbase-user/201505.mbox/%3CCANZDn9ueFEEuZMx=pZdmtLsdGLyZz=rrm1N6EQvLswYc1z-H=g...@mail.gmail.com%3E)
> is exactly what happened in our production env.
> But what the email didn't point out is that the difference between serverName
> in RS and zk node can cause SSH stop to work. as we can see from the code in
> {{RegionServerTracker}}
> {code}
> @Override
> public void nodeDeleted(String path) {
> if (path.startsWith(watcher.rsZNode)) {
> String serverName = ZKUtil.getNodeName(path);
> LOG.info("RegionServer ephemeral node deleted, processing expiration ["
> +
> serverName + "]");
> ServerName sn = ServerName.parseServerName(serverName);
> if (!serverManager.isServerOnline(sn)) {
> LOG.warn(serverName.toString() + " is not online or isn't known to
> the master."+
> "The latter could be caused by a DNS misconfiguration.");
> return;
> }
> remove(sn);
> this.serverManager.expireServer(sn);
> }
> }
> {code}
> The server will not be processed by SSH/ServerCrashProcedure. The regions on
> this server will not been assigned again until master restart or failover.
> I know HBASE-9593 was to fix the issue if RS report to duty and crashed
> before it can put up a zk node. It is a very rare case(And controllable， just
> fix the bug making rs to crash). But The issue I metioned can happened more
> often(and uncontrollable, can't be fixed in HBase, due to DNS, hosts config,
> etc.) and have more severe consequence.
> So here I offer some solutions to discuss:
> 1. Revert HBASE-9593 from all branches, Andrew Purtell has reverted it in
> branch-0.98
> 2. Abort RS if master return a different name, otherwise SSH can't work
> properly
> 3. Master accepts whatever servername reported by RS and don't change it.
> 4.correct the zk node if master return another name( idea from Ted Yu)
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17532) Replace explicit type with diamond operator
[ https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900439#comment-15900439 ] Hudson commented on HBASE-17532: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2631 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2631/]) HBASE-17532 Replaced explicit type with diamond operator (stack: rev b53f354763f96d81ce15d7bded6f1bfd97aee68b) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Pair.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketWriterThread.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/expression/NonLeafExpressionNode.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureSchedulerConcurrency.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentListener.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/SortedCopyOnWriteSet.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/KeyLocker.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRegionLocationFinder.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureSuspended.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/EnabledTableSnapshotHandler.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriterBase.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/OOMERegionServer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduceUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/JvmVersion.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedureControllers.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/ReplicationZKNodeCleaner.java * (edit) hbase-external-blockcache/src/main/java/org/apache/hadoop/hbase/io/hfile/MemcachedBlockCache.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/ConnectionCache.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreEngine.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java * (edit)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900440#comment-15900440 ] Hudson commented on HBASE-17707: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2631 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2631/]) HBASE-17707 New More Accurate Table Skew cost function/generator - (tedyu: rev ee1549cc9778af7124e3c7c6b187a0b124385a90) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17756) We should have better introspection of HFiles
Esteban Gutierrez created HBASE-17756: - Summary: We should have better introspection of HFiles Key: HBASE-17756 URL: https://issues.apache.org/jira/browse/HBASE-17756 Project: HBase Issue Type: Brainstorming Components: HFile Reporter: Esteban Gutierrez [~saint@gmail.com] was suggesting to use DataSketches (https://datasketches.github.io) in order to write additional statistics to the HFiles. This could be used to improve our split decisions, troubleshooting or potentially do other interesting analysis without having to perform full table scans. The statistics could be stored as part of the HFile but we could initially improve the visibility of the data by adding some statistics to HFilePrettyPrinter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17705) Procedure execution must fail fast if procedure is not registered
[ https://issues.apache.org/jira/browse/HBASE-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900432#comment-15900432 ] Ted Yu commented on HBASE-17705: lgtm > Procedure execution must fail fast if procedure is not registered > - > > Key: HBASE-17705 > URL: https://issues.apache.org/jira/browse/HBASE-17705 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17705.v1.patch > > > The issue was discovered during backup testing and described here: > https://issues.apache.org/jira/browse/HBASE-14123?focusedCommentId=15886712=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15886712 > If procedure is not registered, client will be retrying until max attempts > reached. The Master should throw DoNotRetryIOException and client RPC should > handle this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17755) CellBasedKeyBlockIndexReader#midkey should exhaust search of the target middle key on skewed regions
[ https://issues.apache.org/jira/browse/HBASE-17755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900424#comment-15900424 ] Esteban Gutierrez commented on HBASE-17755: --- I worked on this with [~saint@gmail.com] and we should follow up with additional improvements. > CellBasedKeyBlockIndexReader#midkey should exhaust search of the target > middle key on skewed regions > > > Key: HBASE-17755 > URL: https://issues.apache.org/jira/browse/HBASE-17755 > Project: HBase > Issue Type: Bug > Components: HFile >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > > We have always been returning the middle key of the the block index > regardless the distribution of the data on an HFile. A side effect of that > approach is that when millions of rows share the same key its quite easy to > run into a situation when the start key is equal to the middle key or when > the end key is equal to the middle key making that HFile nearly impossible to > split until enough data is written into the region and the middle key shifts > to another row or when an operator uses a custom split point in order to > split that region. > Instead we should exhaust the search of the middle key in the block index in > order to be able to split an HFile earlier when possible even if our edge > case is to serve a region that could hold a single key with millions of > versions of a row or with millions of qualifiers on the same row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17755) CellBasedKeyBlockIndexReader#midkey should exhaust search of the target middle key on skewed regions
Esteban Gutierrez created HBASE-17755: - Summary: CellBasedKeyBlockIndexReader#midkey should exhaust search of the target middle key on skewed regions Key: HBASE-17755 URL: https://issues.apache.org/jira/browse/HBASE-17755 Project: HBase Issue Type: Bug Components: HFile Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez We have always been returning the middle key of the the block index regardless the distribution of the data on an HFile. A side effect of that approach is that when millions of rows share the same key its quite easy to run into a situation when the start key is equal to the middle key or when the end key is equal to the middle key making that HFile nearly impossible to split until enough data is written into the region and the middle key shifts to another row or when an operator uses a custom split point in order to split that region. Instead we should exhaust the search of the middle key in the block index in order to be able to split an HFile earlier when possible even if our edge case is to serve a region that could hold a single key with millions of versions of a row or with millions of qualifiers on the same row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17501) NullPointerException after Datanodes Decommissioned and Terminated
[
https://issues.apache.org/jira/browse/HBASE-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900386#comment-15900386
]
Hadoop QA commented on HBASE-17501:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 5s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 107m 52s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 11s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856302/HBASE_17501.patch.v4 |
| JIRA Issue | HBASE-17501 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0c647853ce11 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ee1549c |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5992/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5992/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> NullPointerException after Datanodes Decommissioned and Terminated
>
[jira] [Commented] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900381#comment-15900381 ] Tianying Chang commented on HBASE-17453: [~stack] sorry for the long delay. What we want is a simple API at RS side that we can use to poll to verify if the connection behave weird, and need reconnect. Earlier, GetPrococolVersion in 94 was borrowed to achieve this goal as a side effect. Basically, this call should be a simple and cheap one. I can made a patch for trunk if needed. > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Attachments: HBASE-17453-1.2.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17719) PreEmptive Fast Fail does not apply to scanners
[ https://issues.apache.org/jira/browse/HBASE-17719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17719: --- Status: Open (was: Patch Available) > PreEmptive Fast Fail does not apply to scanners > --- > > Key: HBASE-17719 > URL: https://issues.apache.org/jira/browse/HBASE-17719 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: James Moore >Assignee: James Moore > Attachments: HBASE_17719.patch > > > on CDH 5.9.0 testing revealed that scanners do not leverage Pre-emptive fast > fail. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900325#comment-15900325 ] Hudson commented on HBASE-17716: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #113 (See [https://builds.apache.org/job/HBase-1.2-JDK7/113/]) HBASE-17716 Formalize Scan Metric names (stack: rev d46e9613b4efd88a82efb437398389a785c0e12b) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ServerSideScanMetrics.java > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Fix For: 2.0.0, 1.3.1, 1.2.5 > > Attachments: HBASE-17716.patch, HBASE-17716_v2.patch, > HBASE-17716_v2.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17754) [C++] RawAsyncTable
Enis Soztutar created HBASE-17754:
-
Summary: [C++] RawAsyncTable
Key: HBASE-17754
URL: https://issues.apache.org/jira/browse/HBASE-17754
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17466) [C++] Test cleanup
[ https://issues.apache.org/jira/browse/HBASE-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-17466: -- Attachment: hbase-17466_v1.patch > [C++] Test cleanup > -- > > Key: HBASE-17466 > URL: https://issues.apache.org/jira/browse/HBASE-17466 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Attachments: hbase-17466_v1.patch > > > The tests takes too long due to sleep and starting / stopping the cluster. We > can do some speed up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17466) [C++] Test cleanup
[ https://issues.apache.org/jira/browse/HBASE-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-17466: -- Summary: [C++] Test cleanup (was: [C++] Speed up the tests a bit) > [C++] Test cleanup > -- > > Key: HBASE-17466 > URL: https://issues.apache.org/jira/browse/HBASE-17466 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > > The tests takes too long due to sleep and starting / stopping the cluster. We > can do some speed up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17466) [C++] Speed up the tests a bit
[ https://issues.apache.org/jira/browse/HBASE-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar reassigned HBASE-17466: - Assignee: Enis Soztutar (was: Sudeep Sunthankar) > [C++] Speed up the tests a bit > -- > > Key: HBASE-17466 > URL: https://issues.apache.org/jira/browse/HBASE-17466 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > > The tests takes too long due to sleep and starting / stopping the cluster. We > can do some speed up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17568) Expire region reports in the HMaster
[ https://issues.apache.org/jira/browse/HBASE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17568: --- Hadoop Flags: Reviewed > Expire region reports in the HMaster > > > Key: HBASE-17568 > URL: https://issues.apache.org/jira/browse/HBASE-17568 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17568.001.patch, > HBASE-17568.002.HBASE-16961.patch, HBASE-17568.003.HBASE-16961.patch > > > (From a TODO) > The RegionServers send reports of sizes from to the Master so the Master can > compute the "size" of each Table. > Presently, the Master has no way to expire these reports. Thus, reports for > tables that are deleted would be retained in memory. Retain the time at which > the report was received by the master, and then use that to determine when to > age it off. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17568) Expire region reports in the HMaster
[ https://issues.apache.org/jira/browse/HBASE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17568: --- Resolution: Fixed Status: Resolved (was: Patch Available) Merging this one over (manually fixing that tab issue). > Expire region reports in the HMaster > > > Key: HBASE-17568 > URL: https://issues.apache.org/jira/browse/HBASE-17568 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17568.001.patch, > HBASE-17568.002.HBASE-16961.patch, HBASE-17568.003.HBASE-16961.patch > > > (From a TODO) > The RegionServers send reports of sizes from to the Master so the Master can > compute the "size" of each Table. > Presently, the Master has no way to expire these reports. Thus, reports for > tables that are deleted would be retained in memory. Retain the time at which > the report was received by the master, and then use that to determine when to > age it off. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17753) Update QuotaObserverChore to include computed snapshot sizes
Josh Elser created HBASE-17753: -- Summary: Update QuotaObserverChore to include computed snapshot sizes Key: HBASE-17753 URL: https://issues.apache.org/jira/browse/HBASE-17753 Project: HBase Issue Type: Sub-task Reporter: Josh Elser Need to update QuotaObserverChore to include the new snapshot size computations that were implemented in HBASE-17749 so that the quota utilizations are accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900266#comment-15900266 ] Hudson commented on HBASE-17716: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #131 (See [https://builds.apache.org/job/HBase-1.3-JDK8/131/]) HBASE-17716 Formalize Scan Metric names (stack: rev d6ca93ce123ac03848d06733201c7d62c24604b2) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ServerSideScanMetrics.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Fix For: 2.0.0, 1.3.1, 1.2.5 > > Attachments: HBASE-17716.patch, HBASE-17716_v2.patch, > HBASE-17716_v2.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-12790) Support fairness across parallelized scans
[
https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900259#comment-15900259
]
Samarth Jain edited comment on HBASE-12790 at 3/7/17 10:07 PM:
---
Thanks for the pointer [~anoop.hbase]. I see that there is already an executor
RWQueueRpcExecutor that gets hold of the ScanRequest in the dispatch method.
{code}
private boolean isScanRequest(final RequestHeader header, final Message param) {
if (param instanceof ScanRequest) {
// The first scan request will be executed as a "short read"
ScanRequest request = (ScanRequest)param;
return request.hasScannerId();
}
return false;
}
@Override
public boolean dispatch(final CallRunner callTask) throws
InterruptedException {
...
if (numScanQueues > 0 && isScanRequest(call.getHeader(), call.param)) {
queueIndex = numWriteQueues + numReadQueues +
scanBalancer.getNextQueue();
}
...
{code}
So yes, there is a way forward by utilizing the scan attribute for this purpose
without having to add an API to Operation. Having said that, looking at the
isWriteRequest method in the same class, I see that things can get
gnarly/brittle/inefficient.
{code}
private boolean isWriteRequest(final RequestHeader header, final Message param)
{
// TODO: Is there a better way to do this?
if (param instanceof MultiRequest) {
MultiRequest multi = (MultiRequest)param;
for (RegionAction regionAction : multi.getRegionActionList()) {
for (Action action: regionAction.getActionList()) {
if (action.hasMutation()) {
return true;
}
}
}
}
{code}
So the ideal would be to have a generic enough API to enable clients mark
read/write requests for whatever they would want to do with it on the server
side.
was (Author: samarthjain):
Thanks for the pointer [~anoop.hbase]. I see that there is already an executor
RWQueueRpcExecutor that gets hold of the ScanRequest in the dispatch method.
{code}
private boolean isScanRequest(final RequestHeader header, final Message param) {
if (param instanceof ScanRequest) {
// The first scan request will be executed as a "short read"
ScanRequest request = (ScanRequest)param;
return request.hasScannerId();
}
return false;
}
@Override
public boolean dispatch(final CallRunner callTask) throws
InterruptedException {
...
if (numScanQueues > 0 && isScanRequest(call.getHeader(), call.param)) {
queueIndex = numWriteQueues + numReadQueues +
scanBalancer.getNextQueue();
}
...
{code}
So yes, there is a way forward by utilizing the scan attribute for this purpose
without having to add an API to Operation. Having said that, looking at the
isWriteRequest method in the same class, I see that things can get
gnarly/brittle/inefficient.
{code}
private boolean isWriteRequest(final RequestHeader header, final Message param)
{
// TODO: Is there a better way to do this?
if (param instanceof MultiRequest) {
MultiRequest multi = (MultiRequest)param;
for (RegionAction regionAction : multi.getRegionActionList()) {
for (Action action: regionAction.getActionList()) {
if (action.hasMutation()) {
return true;
}
}
}
}
{code}
> Support fairness across parallelized scans
> --
>
> Key: HBASE-12790
> URL: https://issues.apache.org/jira/browse/HBASE-12790
> Project: HBase
> Issue Type: New Feature
>Reporter: James Taylor
>Assignee: ramkrishna.s.vasudevan
> Labels: Phoenix
> Attachments: AbstractRoundRobinQueue.java, HBASE-12790_1.patch,
> HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, HBASE-12790.patch,
> HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip
>
>
> Some HBase clients parallelize the execution of a scan to reduce latency in
> getting back results. This can lead to starvation with a loaded cluster and
> interleaved scans, since the RPC queue will be ordered and processed on a
> FIFO basis. For example, if there are two clients, A & B that submit largish
> scans at the same time. Say each scan is broken down into 100 scans by the
> client (broken down into equal depth chunks along the row key), and the 100
> scans of client A are queued first, followed immediately by the 100 scans of
> client B. In this case, client B will be starved out of getting any results
> back until the scans for client A complete.
> One solution to this is to use the attached AbstractRoundRobinQueue instead
> of the standard FIFO queue. The queue to be used could be (maybe it already
> is) configurable based on a new config parameter. Using this queue would
> require the client to have the
[jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
[
https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900259#comment-15900259
]
Samarth Jain commented on HBASE-12790:
--
Thanks for the pointer [~anoop.hbase]. I see that there is already an executor
RWQueueRpcExecutor that gets hold of the ScanRequest in the dispatch method.
{code}
private boolean isScanRequest(final RequestHeader header, final Message param) {
if (param instanceof ScanRequest) {
// The first scan request will be executed as a "short read"
ScanRequest request = (ScanRequest)param;
return request.hasScannerId();
}
return false;
}
@Override
public boolean dispatch(final CallRunner callTask) throws
InterruptedException {
...
if (numScanQueues > 0 && isScanRequest(call.getHeader(), call.param)) {
queueIndex = numWriteQueues + numReadQueues +
scanBalancer.getNextQueue();
}
...
{code}
So yes, there is a way forward by utilizing the scan attribute for this purpose
without having to add an API to Operation. Having said that, looking at the
isWriteRequest method in the same class, I see that things can get
gnarly/brittle/inefficient.
{code}
private boolean isWriteRequest(final RequestHeader header, final Message param)
{
// TODO: Is there a better way to do this?
if (param instanceof MultiRequest) {
MultiRequest multi = (MultiRequest)param;
for (RegionAction regionAction : multi.getRegionActionList()) {
for (Action action: regionAction.getActionList()) {
if (action.hasMutation()) {
return true;
}
}
}
}
{code}
> Support fairness across parallelized scans
> --
>
> Key: HBASE-12790
> URL: https://issues.apache.org/jira/browse/HBASE-12790
> Project: HBase
> Issue Type: New Feature
>Reporter: James Taylor
>Assignee: ramkrishna.s.vasudevan
> Labels: Phoenix
> Attachments: AbstractRoundRobinQueue.java, HBASE-12790_1.patch,
> HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, HBASE-12790.patch,
> HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip
>
>
> Some HBase clients parallelize the execution of a scan to reduce latency in
> getting back results. This can lead to starvation with a loaded cluster and
> interleaved scans, since the RPC queue will be ordered and processed on a
> FIFO basis. For example, if there are two clients, A & B that submit largish
> scans at the same time. Say each scan is broken down into 100 scans by the
> client (broken down into equal depth chunks along the row key), and the 100
> scans of client A are queued first, followed immediately by the 100 scans of
> client B. In this case, client B will be starved out of getting any results
> back until the scans for client A complete.
> One solution to this is to use the attached AbstractRoundRobinQueue instead
> of the standard FIFO queue. The queue to be used could be (maybe it already
> is) configurable based on a new config parameter. Using this queue would
> require the client to have the same identifier for all of the 100 parallel
> scans that represent a single logical scan from the clients point of view.
> With this information, the round robin queue would pick off a task from the
> queue in a round robin fashion (instead of a strictly FIFO manner) to prevent
> starvation over interleaved parallelized scans.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17752) Update reporting RPCs/Shell commands to break out space utilization by snapshot
Josh Elser created HBASE-17752:
--
Summary: Update reporting RPCs/Shell commands to break out space
utilization by snapshot
Key: HBASE-17752
URL: https://issues.apache.org/jira/browse/HBASE-17752
Project: HBase
Issue Type: Sub-task
Reporter: Josh Elser
For adminstrators running HBase with space quotas, it is useful to provide a
breakdown of the utilization of a table. For example, it may be non-intuitive
that a table's utilization is primarily made up of snapshots. We should provide
a new command or modify existing commands such that an admin can see the
utilization for a table/ns:
e.g.
{noformat}
table1: 17GB
resident: 10GB
snapshot_a: 5GB
snapshot_b: 2GB
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17751) Update snapshot manifest to include table it was created from
Josh Elser created HBASE-17751: -- Summary: Update snapshot manifest to include table it was created from Key: HBASE-17751 URL: https://issues.apache.org/jira/browse/HBASE-17751 Project: HBase Issue Type: Sub-task Reporter: Josh Elser In order to tie a snapshot back to a table (and add its utilization to that table's quota), we need to store that lineage of source table to snapshot. Add that information to the snapshot manifest file in a manner which is backwards compatible (e.g. upon upgrade, some snapshots may not have this field -- need to make sure we can handle both cases safely). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17750) Update RS Chore that computes Region sizes to avoid double-counting on rematerialized tables
Josh Elser created HBASE-17750: -- Summary: Update RS Chore that computes Region sizes to avoid double-counting on rematerialized tables Key: HBASE-17750 URL: https://issues.apache.org/jira/browse/HBASE-17750 Project: HBase Issue Type: Sub-task Reporter: Josh Elser When a table is restored from a snapshot, it will reference files that are also referenced by the Snapshot (and potentially the source table). We need to make sure that these restored tables do not also "count" the size of those files as it would make the actual FS utilization incorrect. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17749) Create Master Chore to the size of each Snapshot against a table with a quota
Josh Elser created HBASE-17749: -- Summary: Create Master Chore to the size of each Snapshot against a table with a quota Key: HBASE-17749 URL: https://issues.apache.org/jira/browse/HBASE-17749 Project: HBase Issue Type: Sub-task Reporter: Josh Elser See design doc in the umbrella (HBASE-17748) as to how the size of a snapshot is defined. For each table that has a quota (either set directly or "inherited" from its namespace), we need to compute the size of that snapshot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17748) Include HBase Snapshots in Space Quotas
Josh Elser created HBASE-17748: -- Summary: Include HBase Snapshots in Space Quotas Key: HBASE-17748 URL: https://issues.apache.org/jira/browse/HBASE-17748 Project: HBase Issue Type: Umbrella Reporter: Josh Elser Assignee: Josh Elser Umbrella issue for the inclusion of HBase Snapshots in the Space Quota feature (HBASE-16961) https://docs.google.com/document/d/1f7utThEBYRXYHvp3e5fOhQBv2K1aeuzGHGEfNNE3Clc/edit# / https://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase-Snapshots.pdf -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15941) HBCK repair should not unsplit healthy splitted region
[ https://issues.apache.org/jira/browse/HBASE-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900235#comment-15900235 ] Esteban Gutierrez commented on HBASE-15941: --- Hi [~syuanjiang], can you take a look to this approach? Basically if we are 100% sure that there is parent is present and the splits are valid daughters then we remove the parent and sideline the parent. Otherwise we fallback to the current approach. Thanks! > HBCK repair should not unsplit healthy splitted region > -- > > Key: HBASE-15941 > URL: https://issues.apache.org/jira/browse/HBASE-15941 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 1.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Esteban Gutierrez > Attachments: HBASE-15941.master.001.patch > > > Currently HBCK design in branch-1 has a flaw when repair option (the > -fixHdfsOverlaps option specifically) is specified: it would wrongly merge > split region (by looking at HDFS, it thinks that there exists overlapped > regions - parent region and daughter regions covers the same key range, of > course). See HBASE-15940 for details. > This JIRA tracks the improvement not-to-merge split region in HBCK repair. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900219#comment-15900219 ] Hudson commented on HBASE-17716: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #108 (See [https://builds.apache.org/job/HBase-1.2-JDK8/108/]) HBASE-17716 Formalize Scan Metric names (stack: rev d46e9613b4efd88a82efb437398389a785c0e12b) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ServerSideScanMetrics.java > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Fix For: 2.0.0, 1.3.1, 1.2.5 > > Attachments: HBASE-17716.patch, HBASE-17716_v2.patch, > HBASE-17716_v2.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Attachment: 17728.v8.txt > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt, > 17728.v5.txt, 17728.v6.txt, 17728.v7.txt, 17728.v8.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to create separate build target for Configuration which is > depended upon by both core and security modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17501) NullPointerException after Datanodes Decommissioned and Terminated
[ https://issues.apache.org/jira/browse/HBASE-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900202#comment-15900202 ] James Moore commented on HBASE-17501: - It looks like QA isn't firing the build over in https://builds.apache.org/job/PreCommit-HBASE-Build/ is there a flag that needs to be set for QA to rebuild? > NullPointerException after Datanodes Decommissioned and Terminated > -- > > Key: HBASE-17501 > URL: https://issues.apache.org/jira/browse/HBASE-17501 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Patrick Dignan >Assignee: James Moore >Priority: Minor > Attachments: HBASE_17501.patch, HBASE_17501.patch, > HBASE_17501.patch.v2, HBASE_17501.patch.v3, HBASE_17501.patch.v4, > HBASE_17501.v3 > > > We recently encountered an interesting NullPointerException in HDFS that > bubbles up to HBase, and is resolved be restarting the regionserver. The > issue was exhibited while we were replacing a set of nodes in one of our > clusters with a new set. We did the following: > 1. Turn off the HBase balancer > 2. Gracefully move the regions off the nodes we’re shutting off using a tool > we wrote to do so > 3. Decommission the datanodes using the HDFS exclude hosts file and hdfs > dfsadmin -refreshNodes > 4. Wait for the datanodes to decommission fully > 5. Terminate the VMs the instances are running inside. > A few notes. We did not shutdown the datanode processes, and the nodes were > therefore not marked as dead by the namenode. We simply terminated the > datanode VM (in this case an AWS instance). The nodes were marked as > decommissioned. We are running our clusters with DNS, and when we terminate > VMs, the associated CName is removed and no longer resolves. The errors do > not seem to resolve without a restart. > After we did this, the remaining regionservers started throwing > NullPointerExceptions with the following stack trace: > 2017-01-19 23:09:05,638 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: > RpcServer.RW.fifo.Q.read.handler=80,queue=14,port=60020: callId: 1727723891 > service: ClientService methodName: Scan size: 216 connection: > 172.16.36.128:31538 > java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2214) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:204) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1564) > at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1434) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1682) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1542) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:445) > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:266) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:642) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:592) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:294) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:199) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:343) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:198) > at > org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2106) > at > org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2096) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5544) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2569) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2555) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2536) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2405) > at >
[jira] [Assigned] (HBASE-17725) LimitedPrivate COPROC / PHOENIX interface broken on maintenance release
[
https://issues.apache.org/jira/browse/HBASE-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-17725:
---
Assignee: Sean Busbey
> LimitedPrivate COPROC / PHOENIX interface broken on maintenance release
> ---
>
> Key: HBASE-17725
> URL: https://issues.apache.org/jira/browse/HBASE-17725
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.2.5, 1.1.8
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 1.2.5
>
> Attachments: 1.2.4_1.2.5RC0_compat_report.html
>
>
> The compatibility report for HBase 1.2.5 RC0 flagged one item, the interface
> {{RpcServiceInterface}} added a new method {{setRsRpcServices}}. That
> interface is labeled {{IA.LimitedPrivate(COPROC, PHOENIX)}}.
> AFAICT this change came in on HBASE-16972. There's discussion on that ticket
> about the compatibility impact of the change in logging on earlier releases
> (which I remain fine with). The only discussion about binary/source
> compatibility expressly changes a different part of the patch to avoid
> breaking coprocessors and phoenix.
> As an extra wrinkle, this change has already been published in the 1.1.z
> release line for 1.1.8 and 1.1.9. Apologies, but I haven't gone to verify
> there wasn't a discussion of this break when 1.1.8 came out.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900144#comment-15900144 ] Hudson commented on HBASE-17716: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #120 (See [https://builds.apache.org/job/HBase-1.3-JDK7/120/]) HBASE-17716 Formalize Scan Metric names (stack: rev d6ca93ce123ac03848d06733201c7d62c24604b2) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ServerSideScanMetrics.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Fix For: 2.0.0, 1.3.1, 1.2.5 > > Attachments: HBASE-17716.patch, HBASE-17716_v2.patch, > HBASE-17716_v2.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17716) Formalize Scan Metric names
[ https://issues.apache.org/jira/browse/HBASE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900133#comment-15900133 ] Hudson commented on HBASE-17716: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2630 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2630/]) HBASE-17716 Formalize Scan Metric names (stack: rev fee86ebd7b75e9db1bdbc52355b5dfe58fdfabf7) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/metrics/ServerSideScanMetrics.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java > Formalize Scan Metric names > --- > > Key: HBASE-17716 > URL: https://issues.apache.org/jira/browse/HBASE-17716 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Karan Mehta >Assignee: Karan Mehta >Priority: Minor > Fix For: 2.0.0, 1.3.1, 1.2.5 > > Attachments: HBASE-17716.patch, HBASE-17716_v2.patch, > HBASE-17716_v2.patch > > > HBase provides various metrics through the API's exposed by ScanMetrics > class. > The JIRA PHOENIX-3248 requires them to be surfaced through the Phoenix > Metrics API. Currently these metrics are referred via hard-coded strings, > which are not formal and can break the Phoenix API. Hence we need to refactor > the code to assign enums for these metrics. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15484) Correct the semantic of batch and partial
[ https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900131#comment-15900131 ] Hudson commented on HBASE-15484: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2630 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2630/]) HBASE-15484 Correct the semantic of batch and partial - amend to fix bug (yangzhe1991: rev b635414e8337be7bc14ca6ae605749c35569b4f7) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java > Correct the semantic of batch and partial > - > > Key: HBASE-15484 > URL: https://issues.apache.org/jira/browse/HBASE-15484 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15484.amend.patch, HBASE-15484.branch-1.v01.patch, > HBASE-15484.v05.patch, HBASE-15484.v06.patch, HBASE-15484-v1.patch, > HBASE-15484-v2.patch, HBASE-15484-v3.patch, HBASE-15484-v4.patch > > > Follow-up to HBASE-15325, as discussed, the meaning of setBatch and > setAllowPartialResults should not be same. We should not regard setBatch as > setAllowPartialResults. > Now we deprecated isPartial and use mayHaveMoreCellsInRow. If it returns > false, current Result must be the last one of this row. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17501) NullPointerException after Datanodes Decommissioned and Terminated
[ https://issues.apache.org/jira/browse/HBASE-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Moore updated HBASE-17501: Status: Patch Available (was: Open) > NullPointerException after Datanodes Decommissioned and Terminated > -- > > Key: HBASE-17501 > URL: https://issues.apache.org/jira/browse/HBASE-17501 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Patrick Dignan >Assignee: James Moore >Priority: Minor > Attachments: HBASE_17501.patch, HBASE_17501.patch, > HBASE_17501.patch.v2, HBASE_17501.patch.v3, HBASE_17501.patch.v4, > HBASE_17501.v3 > > > We recently encountered an interesting NullPointerException in HDFS that > bubbles up to HBase, and is resolved be restarting the regionserver. The > issue was exhibited while we were replacing a set of nodes in one of our > clusters with a new set. We did the following: > 1. Turn off the HBase balancer > 2. Gracefully move the regions off the nodes we’re shutting off using a tool > we wrote to do so > 3. Decommission the datanodes using the HDFS exclude hosts file and hdfs > dfsadmin -refreshNodes > 4. Wait for the datanodes to decommission fully > 5. Terminate the VMs the instances are running inside. > A few notes. We did not shutdown the datanode processes, and the nodes were > therefore not marked as dead by the namenode. We simply terminated the > datanode VM (in this case an AWS instance). The nodes were marked as > decommissioned. We are running our clusters with DNS, and when we terminate > VMs, the associated CName is removed and no longer resolves. The errors do > not seem to resolve without a restart. > After we did this, the remaining regionservers started throwing > NullPointerExceptions with the following stack trace: > 2017-01-19 23:09:05,638 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: > RpcServer.RW.fifo.Q.read.handler=80,queue=14,port=60020: callId: 1727723891 > service: ClientService methodName: Scan size: 216 connection: > 172.16.36.128:31538 > java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2214) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:204) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1564) > at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1434) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1682) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1542) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:445) > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:266) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:642) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:592) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:294) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:199) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:343) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:198) > at > org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2106) > at > org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2096) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5544) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2569) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2555) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2536) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2405) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33738) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170) > ... 3 more -- This message was sent by Atlassian JIRA
[jira] [Commented] (HBASE-17532) Replace explicit type with diamond operator
[ https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900118#comment-15900118 ] Jan Hentschel commented on HBASE-17532: --- Thanks for the help and for rebasing the patch. > Replace explicit type with diamond operator > --- > > Key: HBASE-17532 > URL: https://issues.apache.org/jira/browse/HBASE-17532 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17532.master.001.patch, > HBASE-17532.master.002.patch, HBASE-17532.master.002.patch, > HBASE-17532.master.003.patch > > > Because HBase uses Java 8 explicit types can be replaced with the diamond > operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900112#comment-15900112 ] Ted Yu commented on HBASE-17707: Please include the subject of JIRA in the subject of patch. Here is the version I committed: HBASE-17707 New More Accurate Table Skew cost function/generator - adendum ensure testTableSkewConvergesToZero finishes before timeout > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17732) Add CpType template to CoprocessorEnvironment
[ https://issues.apache.org/jira/browse/HBASE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17732: - Status: Open (was: Patch Available) > Add CpType template to CoprocessorEnvironment > - > > Key: HBASE-17732 > URL: https://issues.apache.org/jira/browse/HBASE-17732 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-17732.master.001.patch, > HBASE-17732.master.002.patch > > > Adding template for coprocessor type to CoprocessorEnvironment will: > - get rid of 'instanceOf' checks we do in various coprocessor hosts > - allow us to load only relevant coprocessors in hosts (right now i think > each host loads all coprocs and it's only during execOperation that it checks > if the coproc is of correct type ie XXCoprocessorHost will load XXObserver, > YYObserver, and all others, and in will check in execOperation if coproc is > of type XXObserver and ignore the rest). > - allow sharing of a bunch functions/classes which are currently 99% > duplicate of each other. For eg. CoprocessorOperations, > CoprocessorOperationWithResult, execOperations(). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17732) Add CpType template to CoprocessorEnvironment
[ https://issues.apache.org/jira/browse/HBASE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17732: - Attachment: HBASE-17732.master.002.patch > Add CpType template to CoprocessorEnvironment > - > > Key: HBASE-17732 > URL: https://issues.apache.org/jira/browse/HBASE-17732 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-17732.master.001.patch, > HBASE-17732.master.002.patch > > > Adding template for coprocessor type to CoprocessorEnvironment will: > - get rid of 'instanceOf' checks we do in various coprocessor hosts > - allow us to load only relevant coprocessors in hosts (right now i think > each host loads all coprocs and it's only during execOperation that it checks > if the coproc is of correct type ie XXCoprocessorHost will load XXObserver, > YYObserver, and all others, and in will check in execOperation if coproc is > of type XXObserver and ignore the rest). > - allow sharing of a bunch functions/classes which are currently 99% > duplicate of each other. For eg. CoprocessorOperations, > CoprocessorOperationWithResult, execOperations(). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900044#comment-15900044 ] Karan Mehta commented on HBASE-14925: - Yes, I can take it up. > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: huaxiang sun > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[
https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900036#comment-15900036
]
Hadoop QA commented on HBASE-17707:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 34s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 36s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856639/HBASE-17707-08.patch |
| JIRA Issue | HBASE-17707 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 50f1f972f3a4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fee86eb |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5991/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5991/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5991/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> New More Accurate Table Skew cost function/generator
>
>
> Key: HBASE-17707
> URL: https://issues.apache.org/jira/browse/HBASE-17707
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 1.2.0
> Environment: CentOS
[jira] [Updated] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Attachment: (was: 17728.v7.txt) > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt, > 17728.v5.txt, 17728.v6.txt, 17728.v7.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to create separate build target for Configuration which is > depended upon by both core and security modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17728) Create separate build target for Configuration classes
[ https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17728: --- Attachment: 17728.v7.txt > Create separate build target for Configuration classes > -- > > Key: HBASE-17728 > URL: https://issues.apache.org/jira/browse/HBASE-17728 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt, > 17728.v5.txt, 17728.v6.txt, 17728.v7.txt > > > User is in security module. > When User::isSecurityEnabled() is added, we need to query Configuration for > security setting. > However, this introduces a circular build dependency: > BUILD FAILED: Cycle found: //connection:connection -> //security:security -> > //core:core -> //connection:connection > This issue is to create separate build target for Configuration which is > depended upon by both core and security modules. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17532) Replace explicit type with diamond operator
[ https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17532: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master branch. Thank you for the patch [~Jan Hentschel] > Replace explicit type with diamond operator > --- > > Key: HBASE-17532 > URL: https://issues.apache.org/jira/browse/HBASE-17532 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17532.master.001.patch, > HBASE-17532.master.002.patch, HBASE-17532.master.002.patch, > HBASE-17532.master.003.patch > > > Because HBase uses Java 8 explicit types can be replaced with the diamond > operator. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[ https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899981#comment-15899981 ] Ted Yu commented on HBASE-17747: +1 See if the javadoc warning is related. > Support both weak and soft object pool > -- > > Key: HBASE-17747 > URL: https://issues.apache.org/jira/browse/HBASE-17747 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17747.patch > > > During YCSB testing on embedded mode after HBASE-17744, we found that under > high read load GC is quite severe even with offheap L2 cache. After some > investigation, we found it's caused by using weak reference for > IdReadWriteLock. In embedded mode the read is so quick that the lock might > already get promoted to the old generation when the weak reference is > cleared, which causes dirty card table thus slowing YGC. > So we proposed to use soft reference for this IdReadWriteLock used in cache, > which won't get cleared until JVM memory is not enough, and could resolve the > issue mentioned above. What's more, we propose to extend the WeakObjectPool > to be more generate to support both weak and soft reference. > Note that this issue only emerges under embedded mode with DirectOperator, in > which case all costs on the wire is removed thus produces extremely high > workloads. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17743) Support embedded mode in HBase
[ https://issues.apache.org/jira/browse/HBASE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899978#comment-15899978 ] stack commented on HBASE-17743: --- [~carp84] Looks great. Just to say that work already by-passing RPC in particular when Master was hosting hbase:meta. Let me know if you need pointers. You've seen RegionAsTable class? You talking of an embedded instance running with more than one table? Would it be too radical doing one table per embedded instance? Could then just add a main on RegionAsTable then (smile). > Support embedded mode in HBase > -- > > Key: HBASE-17743 > URL: https://issues.apache.org/jira/browse/HBASE-17743 > Project: HBase > Issue Type: New Feature >Reporter: Yu Li >Assignee: Yu Li > Attachments: Embedded HBase Design.docx > > > Currently HBase supports stand-alone mode and suggest to use it for embedded > database requirements, which has below problems: > 1. HMaster and HRegionServer are still separated threads, which cost more > resource > 2. There’s still RPC cost when doing request, which makes a poor performance > 3. There're still distributed component involved in standalone-mode, such as > zookeeper and DLS, which makes the startup/shutdown speed not acceptable for > embedded usage. > Here we propose to support the embedded mode natively, mainly in two steps: > 1. Simple embedded mode, which still uses zookeeper for coordination, but > with only one server thread and supports access w/o RPC > 2. Full embedded mode, which removes all distributed stuff and make a clean > embedded core for embedded usage. > Please see more details in the design doc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17532) Replace explicit type with diamond operator
[
https://issues.apache.org/jira/browse/HBASE-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899977#comment-15899977
]
Hadoop QA commented on HBASE-17532:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 427 new or modified
test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
5s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 19s
{color} | {color:red} hbase-prefix-tree in master has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 27s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} hbase-hadoop2-compat generated 0 new + 1 unchanged - 1
fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} hbase-prefix-tree in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
|
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[
https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899971#comment-15899971
]
Hadoop QA commented on HBASE-17747:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 15s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hbase-common generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 111m 33s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 165m 22s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856631/HBASE-17747.patch |
| JIRA Issue | HBASE-17747 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b2c7a2798ad8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fee86eb |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| javadoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/5989/artifact/patchprocess/diff-javadoc-javadoc-hbase-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5989/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
