[jira] [Commented] (HBASE-16670) Make RpcServer#processRequest logic more robust
[
https://issues.apache.org/jira/browse/HBASE-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512286#comment-15512286
]
Yu Li commented on HBASE-16670:
---
I found this incomplete logic when developing async table codes, some mistake
lead me to some logic like calling {{getStub().mutate(controller, null);}}. So
in our current codes there won't be such a kind of request, this is why I set
the priority of this JIRA to Minor, but I think this is still something worth
improving. Agree?:-)
> Make RpcServer#processRequest logic more robust
> ---
>
> Key: HBASE-16670
> URL: https://issues.apache.org/jira/browse/HBASE-16670
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Attachments: HBASE-16670.patch
>
>
> Currently in {{RpcServer#processRequest}}, we will check whether the request
> header has parameters but missed handling the abnormal case, so if there's no
> param in the header, it will throw NPE like below:
> {noformat}
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2269)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2211)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512263#comment-15512263 ] Hudson commented on HBASE-14734: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #26 (See [https://builds.apache.org/job/HBase-1.2-JDK8/26/]) HBASE-14734 Prevent BindException when setting up MiniKdc. Port for kdc (appy: rev 79531fd95f1b7b72ad1f764d7cd57e37467a4b2a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestGenerateDelegationToken.java > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16651) LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache.
[ https://issues.apache.org/jira/browse/HBASE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16651: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review Ram. > LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache. > - > > Key: HBASE-16651 > URL: https://issues.apache.org/jira/browse/HBASE-16651 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16651.patch > > > In case of L1 and L2 cache usage with combinedMode = false, L2 is used as a > victim handler cache for L1 cache. When a getBlock() request comes, L1 will > see if block is in it and if not it will try to provide the block from L2 > cache. In such a case, the return block must return the block to L2 cache and > count down the ref count for the block. But right now we just ignore the > returnBlock call in LRUCache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16651) LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache.
[ https://issues.apache.org/jira/browse/HBASE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512256#comment-15512256 ] ramkrishna.s.vasudevan commented on HBASE-16651: +1 > LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache. > - > > Key: HBASE-16651 > URL: https://issues.apache.org/jira/browse/HBASE-16651 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16651.patch > > > In case of L1 and L2 cache usage with combinedMode = false, L2 is used as a > victim handler cache for L1 cache. When a getBlock() request comes, L1 will > see if block is in it and if not it will try to provide the block from L2 > cache. In such a case, the return block must return the block to L2 cache and > count down the ref count for the block. But right now we just ignore the > returnBlock call in LRUCache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[ https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16650: --- Attachment: HBASE-16650_V2.patch > Wrong usage of BlockCache eviction stat for heap memory tuning > -- > > Key: HBASE-16650 > URL: https://issues.apache.org/jira/browse/HBASE-16650 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16650.patch, HBASE-16650_V2.patch > > > 1. We use the stat evictedBlocksCount - A block can get evicted because of > eviction thread due to lack of space or because of removal of an HFile itself > (After a compaction). We should not consider the latter in the tune decision > at all. These are actually invalidation of blocks. Should the stat counter > itself not use this count of evicted blocks? I think yes. This will give > wrong message to users that there are lot of real eviction happening. > 2. In case L1+ L2 combined block cache, what we use is the sum of evictions > from both. But we will be tuning L1 size alone. Eviction count from L2 should > not affect the tuning of L1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512231#comment-15512231 ] Ashish Singhi commented on HBASE-16680: --- +1 > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch, HBASE-16680-master_v2.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512227#comment-15512227 ] binlijin commented on HBASE-16680: -- HBASE-16680-master_v2.patch fix it. > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch, HBASE-16680-master_v2.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512226#comment-15512226 ] Ashish Singhi commented on HBASE-16676: --- I am +1 to commit this in branch-1.2 > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16680: - Attachment: HBASE-16680-master_v2.patch > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch, HBASE-16680-master_v2.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1551#comment-1551 ] Sean Busbey commented on HBASE-16676: - it seems silly to not include the fix in branch-1.2 if many of the deployments our committers are involved with using 1.2.z are going patch for the issue. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512213#comment-15512213 ] Ted Yu commented on HBASE-16672: SecureBulkLoadListener has logic for the copy through staging dir. I think we should reuse this part of code. > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, > 16672.v5.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512212#comment-15512212
]
Anoop Sam John commented on HBASE-16650:
bq.We are setting some flags in this block that are used later. Do we need
those flags when an eviction because an hfile has been removed?
No there are no flags being set inside the if and used outside..
Code before patch
{code}
stats.evicted(block.getCachedTime(), block.getCacheKey().isPrimary());
if (evictedByEvictionProcess && victimHandler != null) {
if (victimHandler instanceof BucketCache) {
boolean wait = getCurrentSize() < acceptableSize();
boolean inMemory = block.getPriority() == BlockPriority.MEMORY;
((BucketCache)victimHandler).cacheBlockWithWait(block.getCacheKey(),
block.getBuffer(),
inMemory, wait);
} else {
victimHandler.cacheBlock(block.getCacheKey(), block.getBuffer());
}
}
{code}
Code after this patch
{code}
if (evictedByEvictionProcess ) {
stats.evicted(block.getCachedTime(), block.getCacheKey().isPrimary());
if(victimHandler != null){
if (victimHandler instanceof BucketCache) {
boolean wait = getCurrentSize() < acceptableSize();
boolean inMemory = block.getPriority() == BlockPriority.MEMORY;
((BucketCache)victimHandler).cacheBlockWithWait(block.getCacheKey(),
block.getBuffer(),
inMemory, wait);
} else {
victimHandler.cacheBlock(block.getCacheKey(), block.getBuffer());
}
}
}
{code}
Will commit with addressing ur 1st comment
> Wrong usage of BlockCache eviction stat for heap memory tuning
> --
>
> Key: HBASE-16650
> URL: https://issues.apache.org/jira/browse/HBASE-16650
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16650.patch
>
>
> 1. We use the stat evictedBlocksCount - A block can get evicted because of
> eviction thread due to lack of space or because of removal of an HFile itself
> (After a compaction). We should not consider the latter in the tune decision
> at all. These are actually invalidation of blocks. Should the stat counter
> itself not use this count of evicted blocks? I think yes. This will give
> wrong message to users that there are lot of real eviction happening.
> 2. In case L1+ L2 combined block cache, what we use is the sum of evictions
> from both. But we will be tuning L1 size alone. Eviction count from L2 should
> not affect the tuning of L1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512207#comment-15512207 ] Andrew Purtell commented on HBASE-16676: Sorry I missed the discussion on the other issue. We will definitely patch for this if moving onto 1.2 based code given how it will impact us. Chasing this down after seeing ITBLL failures with 1.2 burned a lot of time for me. Related, I can report a decent probability this will fail ITBLL if you use clusterdock as it is now to test. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16651) LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache.
[ https://issues.apache.org/jira/browse/HBASE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512199#comment-15512199 ] Anoop Sam John commented on HBASE-16651: [~ram_krish] does ur concerns being answered above? Can I get a +1 pls? > LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache. > - > > Key: HBASE-16651 > URL: https://issues.apache.org/jira/browse/HBASE-16651 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16651.patch > > > In case of L1 and L2 cache usage with combinedMode = false, L2 is used as a > victim handler cache for L1 cache. When a getBlock() request comes, L1 will > see if block is in it and if not it will try to provide the block from L2 > cache. In such a case, the return block must return the block to L2 cache and > count down the ref count for the block. But right now we just ignore the > returnBlock call in LRUCache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512187#comment-15512187 ] Ashish Singhi commented on HBASE-16676: --- I think we should bring this change in branch-1.2 also. Even our customers reported this and unfortunately we are maintaining this as a private change in our version. As at that time it was decided that it will not be committed in branch-1.2. Committing it in branch-1.2 will help many of us and IMO this seems more like a bug fix! > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16666) Add append and remove peer namespaces cmds for replication
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-1: --- Attachment: HBASE-1-v1.patch Attach v1. Failed ut Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.minikdc.MiniKdc. Update hbase-shell pom for minikdc. > Add append and remove peer namespaces cmds for replication > -- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-1-v1.patch, HBASE-1.patch > > > After HBASE-16447, we support replication by namespaces config in peer. Like > append_peer_tableCFs and remove_peer_tableCFs, I thought we need two new > shell cmd: append_peer_namespaces and remove_peer_namespaces. Then we can > easily change the namespaces config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512172#comment-15512172 ] Anoop Sam John commented on HBASE-16672: The copy of the backup files should be done by the restore op and then pass the copied file path to the bulk load rather than giving option for the bulk load itself to rename/copy? > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, > 16672.v5.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[
https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512174#comment-15512174
]
Duo Zhang commented on HBASE-16679:
---
The {{maxThroughputPerOperation}} should be volatile.
And the {{HBaseTestingUtility}} does not need to be static as you recreate it
in setUp?
Thanks.
> Flush throughput controller: Minor perf change and fix flaky
> TestFlushWithThroughputController
> --
>
> Key: HBASE-16679
> URL: https://issues.apache.org/jira/browse/HBASE-16679
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16679.master.001.patch,
> HBASE-16679.master.002.patch
>
>
> Minor perf change:
> Calculate maxThroughputPerOperation outside of control() since start()()
> are called only once per operation, but control can be called
> hundreds/thousands of time.
> Flaky test:
> Problems in current test:
> - writes only 2.5MB each iteration but control triggers sleep only every 1Mb
> write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL).
> Either increase data written in each batch or decreasing this threshold for
> better throughput control.
> - We shouldn't be timing table disable/delete/create and populating data in
> throughput calculations.
> See the differences below.
> With patch (total data written 30M)
> run 1:
> Throughput is: 1.0113841089709052 MB/s
> Throughput w/o limit is: 14.665069580078125 MB/s
> With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms
> run 2:
> Throughput is: 1.0113841089709052 MB/s
> Throughput w/o limit is: 14.665069580078125 MB/s
> With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms
> Without patch (total data written 25M)
> run 1:
> Throughput is: 0.921681903523776 MB/s
> Throughput w/o limit is: 4.06833346870301 MB/s
> With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms
> run 2:
> Throughput is: 0.9422982728478803 MB/s
> Throughput w/o limit is: 4.047858424942981 MB/s
> With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512170#comment-15512170 ] Sean Busbey commented on HBASE-16676: - no worries [~ashish singhi]. do you think the impact doesn't warrant the change in behavior? > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512161#comment-15512161 ] Ashish Singhi commented on HBASE-16676: --- Sorry, I did not refresh the page and missed Sean's comment. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512151#comment-15512151 ] Ashish Singhi commented on HBASE-16676: --- The discussion regarding this was happened on HBASE-15315 and was decided to not commit in branch-1.2 as it will introduce a behavior change. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16670) Make RpcServer#processRequest logic more robust
[
https://issues.apache.org/jira/browse/HBASE-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512149#comment-15512149
]
Anoop Sam John commented on HBASE-16670:
So how u r getting a requesting of this type?
> Make RpcServer#processRequest logic more robust
> ---
>
> Key: HBASE-16670
> URL: https://issues.apache.org/jira/browse/HBASE-16670
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Attachments: HBASE-16670.patch
>
>
> Currently in {{RpcServer#processRequest}}, we will check whether the request
> header has parameters but missed handling the abnormal case, so if there's no
> param in the header, it will throw NPE like below:
> {noformat}
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2269)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2211)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512141#comment-15512141 ] Sean Busbey commented on HBASE-16676: - This is effectively a backport of HBASE-15315, right? The original reasoning for that not ending up in branch-1.2 was that it would change operational behavior too much for a maintenance release. It sounds like another ~6 months of time on the 1.2 branch has led to a belief that the downside of leaving this in place is severe enough to change that decision. Is that right? If so, that sounds reasonable to me. ping [~eclark] since he was the other person in favor of leaving HBASE-15315 out of branch-1.2. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[
https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512135#comment-15512135
]
Hadoop QA commented on HBASE-16676:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 28m 48s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
19s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 43s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 118m 18s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 192m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.regionserver.TestWALLockup |
| | hadoop.hbase.regionserver.TestHRegion |
| | hadoop.hbase.coprocessor.TestRowProcessorEndpoint |
| | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-22 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829735/HBASE-16676-branch-1.2.patch
|
| JIRA Issue | HBASE-16676 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5b559c633c4d
[jira] [Commented] (HBASE-16680) Reduce garbage in BufferChain
[
https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512129#comment-15512129
]
Ashish Singhi commented on HBASE-16680:
---
bq. for (int i = 0; i < cellBlock.size(); i++) {
Nit: extract cellBlock.size() to a variable and use it.
> Reduce garbage in BufferChain
> -
>
> Key: HBASE-16680
> URL: https://issues.apache.org/jira/browse/HBASE-16680
> Project: HBase
> Issue Type: Improvement
>Reporter: binlijin
>Assignee: binlijin
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16680-master.patch
>
>
> BufferChain accept a List and then convert it to ByteBuffer[], we
> can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate
> the object List.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512123#comment-15512123 ] Hudson commented on HBASE-14734: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #18 (See [https://builds.apache.org/job/HBase-1.3-JDK7/18/]) HBASE-14734 Prevent BindException when setting up MiniKdc. Port for kdc (appy: rev 1c42948d5ec691b1f27b5c61f5589fdf59e16b6c) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/AbstractTestSecureIPC.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/SecureTestCluster.java > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
Appy created HBASE-16679: Summary: Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController Key: HBASE-16679 URL: https://issues.apache.org/jira/browse/HBASE-16679 Project: HBase Issue Type: Bug Reporter: Appy Assignee: Appy Minor perf change: Calculate maxThroughputPerOperation outside of control() since start()() are called only once per operation, but control can be called hundreds/thousands of time. Flaky test: Problems in current test: - writes only 2.5MB each iteration but control triggers sleep only every 1Mb write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). Either increase data written in each batch or decreasing this threshold for better throughput control. - We shouldn't be timing table disable/delete/create and populating data in throughput calculations. See the differences below. With patch (total data written 30M) run 1: Throughput is: 1.0113841089709052 MB/s Throughput w/o limit is: 14.665069580078125 MB/s With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms run 2: Throughput is: 1.0113841089709052 MB/s Throughput w/o limit is: 14.665069580078125 MB/s With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms Without patch (total data written 25M) run 1: Throughput is: 0.921681903523776 MB/s Throughput w/o limit is: 4.06833346870301 MB/s With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms run 2: Throughput is: 0.9422982728478803 MB/s Throughput w/o limit is: 4.047858424942981 MB/s With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16675) Average region size may be incorrect when there is region whose RegionLoad cannot be retrieved
[ https://issues.apache.org/jira/browse/HBASE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512022#comment-15512022 ] Heng Chen commented on HBASE-16675: --- +1 > Average region size may be incorrect when there is region whose RegionLoad > cannot be retrieved > -- > > Key: HBASE-16675 > URL: https://issues.apache.org/jira/browse/HBASE-16675 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16675.v1.txt > > > HBASE-15933 fixed the NullPointerException bug. > When there is one or more region whose RegionLoad cannot be retrieved, the > average region size may be incorrect. > We should not use tableRegions.size() as denominator - the number of regions > whose RegionLoad can be retrieved should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16669) fix flaky TestAdmin#testmergeRegions
[
https://issues.apache.org/jira/browse/HBASE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-16669:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the review Appy. Test only change and other unittests that failed
are unrelated.
> fix flaky TestAdmin#testmergeRegions
>
>
> Key: HBASE-16669
> URL: https://issues.apache.org/jira/browse/HBASE-16669
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16669.v1.patch
>
>
> Recent test runs show that this test is failing with these error messages:
> {code}
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1385)
> {code}
> or
> {code}
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1394)
> {code}
> looking at the code this indicates that merge operation did not complete or
> didn't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512001#comment-15512001
]
Hadoop QA commented on HBASE-16672:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 10s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 175m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829723/16672.v4.txt |
| JIRA Issue |
[jira] [Commented] (HBASE-16666) Add append and remove peer namespaces cmds for replication
[
https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511978#comment-15511978
]
Hadoop QA commented on HBASE-1:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 40s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 11s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 11s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
48m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 25s {color}
| {color:red} hbase-shell in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 9s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestReplicationShell |
| | hadoop.hbase.client.rsgroup.TestShellRSGroups |
| | hadoop.hbase.client.TestShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829544/HBASE-1.patch |
| JIRA Issue | HBASE-1 |
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux b0560da77c1e 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 593fb75 |
| Default Java | 1.8.0_101 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3646/artifact/patchprocess/patch-unit-hbase-shell.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3646/artifact/patchprocess/patch-unit-hbase-shell.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3646/testReport/ |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3646/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add append and remove peer namespaces cmds for replication
> --
>
> Key: HBASE-1
> URL: https://issues.apache.org/jira/browse/HBASE-1
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-1.patch
>
>
> After HBASE-16447, we support replication by namespaces config in peer. Like
> append_peer_tableCFs and remove_peer_tableCFs, I thought we need two new
> shell cmd:
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417.v2.txt Patch v2 adds second full back up in the test. Not working yet. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt, 14417.v2.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511930#comment-15511930
]
Hudson commented on HBASE-16294:
FAILURE: Integrated in Jenkins build HBase-1.4 #421 (See
[https://builds.apache.org/job/HBase-1.4/421/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev 23c5ea39bd237c2012c0c5a13951b8338dc54a8d)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511931#comment-15511931 ] Hudson commented on HBASE-14734: FAILURE: Integrated in Jenkins build HBase-1.4 #421 (See [https://builds.apache.org/job/HBase-1.4/421/]) HBASE-14734 Prevent BindException when setting up MiniKdc. Port for kdc (appy: rev e7e660d5b2b6ed915ec7614e8d7070f9a51d091f) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/AbstractTestSecureIPC.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/SecureTestCluster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12949) Scanner can be stuck in infinite loop if the HFile is corrupted
[
https://issues.apache.org/jira/browse/HBASE-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511932#comment-15511932
]
Hudson commented on HBASE-12949:
FAILURE: Integrated in Jenkins build HBase-1.4 #421 (See
[https://builds.apache.org/job/HBase-1.4/421/])
HBASE-12949 Scanner can be stuck in infinite loop if the HFile is (jerryjch:
rev c80d671a062737c806d4e20a992443e9b6b86b02)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> Scanner can be stuck in infinite loop if the HFile is corrupted
> ---
>
> Key: HBASE-12949
> URL: https://issues.apache.org/jira/browse/HBASE-12949
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3, 0.98.10
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-12949-branch-1-v3.patch, HBASE-12949-master-v2
> (1).patch, HBASE-12949-master-v2.patch, HBASE-12949-master-v2.patch,
> HBASE-12949-master-v2.patch, HBASE-12949-master-v3.patch,
> HBASE-12949-master.patch
>
>
> We've encountered problem where compaction hangs and never completes.
> After looking into it further, we found that the compaction scanner was stuck
> in a infinite loop. See stack below.
> {noformat}
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:296)
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257)
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697)
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:672)
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223)
> {noformat}
> We identified the hfile that seems to be corrupted. Using HFile tool shows
> the following:
> {noformat}
> [biadmin@hdtest009 bin]$ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -k
> -m -f
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> 15/01/23 11:53:17 INFO Configuration.deprecation: hadoop.native.lib is
> deprecated. Instead, use io.native.lib.available
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> 15/01/23 11:53:18 INFO Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> Scanning ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> WARNING, previous row is greater then current row
> filename ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> previous ->
> \x00/20110203-094231205-79442793-1410161293068203000\x0Aattributes16794406\x00\x00\x01\x00\x00\x00\x00\x00\x00
> current ->
> Exception in thread "main" java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:489)
> at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:347)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:856)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:768)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:362)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:262)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:220)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:539)
> at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:802)
> {noformat}
> Turning on Java Assert shows the following:
> {noformat}
> Exception in thread "main" java.lang.AssertionError: Key
> 20110203-094231205-79442793-1410161293068203000/attributes:16794406/1099511627776/Minimum/vlen=15/mvcc=0
> followed by a smaller key //0/Minimum/vlen=0/mvcc=0 in cf attributes
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.checkScanOrder(StoreScanner.java:672)
> {noformat}
> It shows that the hfile seems to be corrupted -- the keys don't seem to be
> right.
> But Scanner is not able to give a meaningful error, but stuck in an infinite
> loop in here:
> {code}
> KeyValueHeap.generalizedSeek()
> while ((scanner = heap.poll()) != null) {
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511790#comment-15511790 ] Enis Soztutar commented on HBASE-12088: --- +1. > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > Attachments: hbase-12088.v0.patch > > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16673) Enhance history command: history per backup set
[
https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511791#comment-15511791
]
Hadoop QA commented on HBASE-16673:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s {color}
| {color:red} HBASE-16673 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829734/HBASE-16673-v1.patch |
| JIRA Issue | HBASE-16673 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3643/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Enhance history command: history per backup set
> ---
>
> Key: HBASE-16673
> URL: https://issues.apache.org/jira/browse/HBASE-16673
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16673-v1.patch
>
>
> New command-line option: -set setName. Will retrieve backup history for a
> particular backup set
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511785#comment-15511785
]
Hudson commented on HBASE-16294:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #18 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/18/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev 9aaff6045b151518a0cdaf694ebd477bff30b2c0)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
[
https://issues.apache.org/jira/browse/HBASE-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511782#comment-15511782
]
Enis Soztutar commented on HBASE-16678:
---
Patch is pretty simple:
{code}
diff --git
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
index 4349537..748d2be 100644
---
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
+++
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
@@ -268,8 +268,9 @@ public class TableRecordReaderImpl {
* @throws IOException
*/
private void updateCounters() throws IOException {
-ScanMetrics scanMetrics = this.scan.getScanMetrics();
+ScanMetrics scanMetrics = currentScan.getScanMetrics();
{code}
Let me see whether we need a unit test.
> MapReduce jobs do not update counters from ScanMetrics
> --
>
> Key: HBASE-16678
> URL: https://issues.apache.org/jira/browse/HBASE-16678
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> Was inspecting a perf issue, where we needed the scanner metrics as counters
> for a MR job. Turns out that the HBase scan counters are no longer working in
> 1.0+. I think it got broken via HBASE-13030.
> These are the counters:
> {code}
> HBase Counters
> BYTES_IN_REMOTE_RESULTS=0
> BYTES_IN_RESULTS=280
> MILLIS_BETWEEN_NEXTS=11
> NOT_SERVING_REGION_EXCEPTION=0
> NUM_SCANNER_RESTARTS=0
> NUM_SCAN_RESULTS_STALE=0
> REGIONS_SCANNED=1
> REMOTE_RPC_CALLS=0
> REMOTE_RPC_RETRIES=0
> RPC_CALLS=3
> RPC_RETRIES=0
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
Enis Soztutar created HBASE-16678:
-
Summary: MapReduce jobs do not update counters from ScanMetrics
Key: HBASE-16678
URL: https://issues.apache.org/jira/browse/HBASE-16678
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
Was inspecting a perf issue, where we needed the scanner metrics as counters
for a MR job. Turns out that the HBase scan counters are no longer working in
1.0+. I think it got broken via HBASE-13030.
These are the counters:
{code}
HBase Counters
BYTES_IN_REMOTE_RESULTS=0
BYTES_IN_RESULTS=280
MILLIS_BETWEEN_NEXTS=11
NOT_SERVING_REGION_EXCEPTION=0
NUM_SCANNER_RESTARTS=0
NUM_SCAN_RESULTS_STALE=0
REGIONS_SCANNED=1
REMOTE_RPC_CALLS=0
REMOTE_RPC_RETRIES=0
RPC_CALLS=3
RPC_RETRIES=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511738#comment-15511738 ] Andrew Purtell commented on HBASE-16662: Got sidetracked. I can commit tomorrow unless someone else wants to get to it first. > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511733#comment-15511733 ] Andrew Purtell commented on HBASE-16676: Hey [~busbey] I have something for you to look at if you have a sec > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Attachment: HBASE-16676-branch-1.2.patch Here's a patch for branch-1.2 that reverts only the behavioral change leaving the LimitedPrivate API changes in place for coprocessor compatibility. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Assignee: Andrew Purtell Fix Version/s: 1.2.4 Status: Patch Available (was: Open) > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.3, 1.2.2, 1.2.1, 1.2.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511718#comment-15511718
]
Ted Yu commented on HBASE-16672:
w.r.t. the SecureBulkLoadListener, we have this code:
{code}
if (bulkLoadListener != null) {
finalPath = bulkLoadListener.prepareBulkLoad(familyName, path);
}
Path commitedStoreFile = store.bulkLoadHFile(finalPath, seqId,
copyFile);
{code}
The new flag would be effective based on the return value from
bulkLoadListener.prepareBulkLoad().
I feel there is no need to modify BulkLoadListener.
Let me know if I missed something.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> This is related to HBASE-14417, to support incrementally restoring to
> multiple destinations, this issue adds option which would always copy
> hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Status: Patch Available (was: Open) > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16656-v1.patch > > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16673: -- Attachment: HBASE-16673-v1.patch patch v1. cc; [~tedyu] > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16673-v1.patch > > > New command-line option: -set setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16673: -- Status: Patch Available (was: Open) > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16673-v1.patch > > > New command-line option: -set setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511698#comment-15511698 ] Ted Yu commented on HBASE-14417: I was adding new test which does one more full table backup after the incremental restore and verifies that BulkLoadDescriptor's are cleaned up from hbase:backup. It turns out the bulk loaded hfiles were renamed during the incremental restore which resulted in FileNotFoundException. Filed HBASE-16672 so that the bulk loaded hfiles can be used for multiple restore destinations. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511700#comment-15511700 ] Andrew Purtell commented on HBASE-16676: That functionality of HBASE-13375 only exists in branch-1.2 . > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Affects Version/s: (was: 1.4.0) (was: 1.3.0) > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Affects Version/s: (was: 2.0.0) > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0, 1.2.1, 1.4.0, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511681#comment-15511681 ] Ted Yu commented on HBASE-16672: I agree that there is no need to dive into the details of the aforementioned stack trace. I simplified the description accordingly. Discussion on the stack trace can be carried out over in HBASE-14417. > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Description: This is related to HBASE-14417, to support incrementally
restoring to multiple destinations, this issue adds option which would always
copy hfile(s) during bulk load. (was: While working on HBASE-14417, I found
that taking full backup of table which received bulk loaded hfiles and later
gone through incremental restore failed with the following exception:
{code}
2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
subprocedure-pool7-thread-1]
snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
Exception in SnapshotSubprocedurePool
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
does not exist:
hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
{code}
The cause was that the bulk loaded hfiles in source table were renamed during
bulk load step of incremental restore.
To support incrementally restoring to multiple destinations, this issue adds
option which would always copy hfile(s) during bulk load.)
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> This is related to HBASE-14417, to support incrementally restoring to
> multiple destinations, this issue adds option which would always copy
> hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16671) Split TestExportSnapshot
[
https://issues.apache.org/jira/browse/HBASE-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511663#comment-15511663
]
Hadoop QA commented on HBASE-16671:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 3s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 57s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 166m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| | hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
| | hadoop.hbase.client.TestAdmin1 |
| | hadoop.hbase.regionserver.TestRegionServerMetrics |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829697/HBASE-16671-v0.patch |
| JIRA Issue | HBASE-16671 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d340db98720d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6a4c292 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3639/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3639/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3639/testReport/ |
| modules | C:
[jira] [Updated] (HBASE-16677) Add table size (total store file size) to table page
[ https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guang Yang updated HBASE-16677: --- Attachment: HBASE-16677_v0.patch > Add table size (total store file size) to table page > > > Key: HBASE-16677 > URL: https://issues.apache.org/jira/browse/HBASE-16677 > Project: HBase > Issue Type: New Feature > Components: website >Reporter: Guang Yang >Priority: Minor > Attachments: HBASE-16677_v0.patch > > > Currently there is not an easy way to get the table size from the web UI, > though we have the region size on the page, it is still convenient to have a > table for the table size stat. > Another pain point is that when the table grow large with tens of thousands > of regions, it took extremely long time to load the page, however, sometimes > we don't want to check all the regions. An optimization could be to accept a > query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511653#comment-15511653
]
Jerry He commented on HBASE-16672:
--
Also there is another place that needs to be taken care of, in the
SecureBulkLoadListener where we move or copy the files to staging dir. In the
secure bulk load case, that is the place to decide copy or rename.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511652#comment-15511652
]
Hudson commented on HBASE-16294:
FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1784 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1784/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev f7961ea70a040d450c7b1393792e43eba076f752)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16677) Add table size (total store file size) to table page
Guang Yang created HBASE-16677: -- Summary: Add table size (total store file size) to table page Key: HBASE-16677 URL: https://issues.apache.org/jira/browse/HBASE-16677 Project: HBase Issue Type: New Feature Components: website Reporter: Guang Yang Priority: Minor Currently there is not an easy way to get the table size from the web UI, though we have the region size on the page, it is still convenient to have a table for the table size stat. Another pain point is that when the table grow large with tens of thousands of regions, it took extremely long time to load the page, however, sometimes we don't want to check all the regions. An optimization could be to accept a query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511645#comment-15511645
]
Jerry He commented on HBASE-16672:
--
Hi, Ted
The stack trace looks confusing.
1. Why incremental restore was bulk loading the file paths that are in an
existing table? The incremental restore should be restoring files from the
backup image. The files in the backup image may be coming from the existing
table originally. But the files should be physically in the backup image. This
is different from the snapshot restore, where it is a in-place restore.
2. What the snapshot failed with the missing file?
An bulk load option to keep the original files (not move/rename them to target)
seems a fine option. We can simplify the JIRA description to describe just
that, without the confusing stack trace.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Attachment: 16672.v4.txt
Patch v4 addresses TestHRegionServerBulkLoadWithOldClient.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511605#comment-15511605 ] Andrew Purtell commented on HBASE-16662: +1 Those test failures do not look related. Let me poke around with this a bit, and if it looks good I will commit with backport to all active branches. > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14734: - Resolution: Fixed Status: Resolved (was: Patch Available) > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
Andrew Purtell created HBASE-16676: -- Summary: All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375 Key: HBASE-16676 URL: https://issues.apache.org/jira/browse/HBASE-16676 Project: HBase Issue Type: Bug Affects Versions: 1.2.3, 1.2.2, 1.2.1, 1.2.0, 2.0.0, 1.3.0, 1.4.0 Reporter: Andrew Purtell I have been trying to track down why 1.2.x won't sometimes pass a 1 billion row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC prioritization could explain it. We get stuck during the loading phase and the loader job eventually fails. All testing is done in an insecure environment under the same UNIX user (clusterdock) so effectively all ops are issued by the superuser. Doing unrelated work - or so I thought! - I was looking at object allocations by YCSB workload by thread and when looking at the RegionServer RPC threads noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 and up, instead the vast majority are from threads named "PriorityRpcServer.handler*" with very little from threads named "B.defaultRpcServer.handler*". A git bisect to find the change that causes this leads to HBASE-13375, and so of course this makes sense out of what I am seeing, but is this really what we want? What about production environments (insecure and degenerate secure) where all ops are effectively issued by the superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14734: - Fix Version/s: 1.2.4 1.4.0 1.3.0 2.0.. > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12949) Scanner can be stuck in infinite loop if the HFile is corrupted
[
https://issues.apache.org/jira/browse/HBASE-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511569#comment-15511569
]
Hudson commented on HBASE-12949:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1649 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1649/])
HBASE-12949 Scanner can be stuck in infinite loop if the HFile is (jerryjch:
rev b6b01a790f600d4699450e2e97349599c9efc0e1)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
> Scanner can be stuck in infinite loop if the HFile is corrupted
> ---
>
> Key: HBASE-12949
> URL: https://issues.apache.org/jira/browse/HBASE-12949
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3, 0.98.10
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-12949-branch-1-v3.patch, HBASE-12949-master-v2
> (1).patch, HBASE-12949-master-v2.patch, HBASE-12949-master-v2.patch,
> HBASE-12949-master-v2.patch, HBASE-12949-master-v3.patch,
> HBASE-12949-master.patch
>
>
> We've encountered problem where compaction hangs and never completes.
> After looking into it further, we found that the compaction scanner was stuck
> in a infinite loop. See stack below.
> {noformat}
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:296)
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257)
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697)
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:672)
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223)
> {noformat}
> We identified the hfile that seems to be corrupted. Using HFile tool shows
> the following:
> {noformat}
> [biadmin@hdtest009 bin]$ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -k
> -m -f
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> 15/01/23 11:53:17 INFO Configuration.deprecation: hadoop.native.lib is
> deprecated. Instead, use io.native.lib.available
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> 15/01/23 11:53:18 INFO Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> Scanning ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> WARNING, previous row is greater then current row
> filename ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> previous ->
> \x00/20110203-094231205-79442793-1410161293068203000\x0Aattributes16794406\x00\x00\x01\x00\x00\x00\x00\x00\x00
> current ->
> Exception in thread "main" java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:489)
> at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:347)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:856)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:768)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:362)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:262)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:220)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:539)
> at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:802)
> {noformat}
> Turning on Java Assert shows the following:
> {noformat}
> Exception in thread "main" java.lang.AssertionError: Key
> 20110203-094231205-79442793-1410161293068203000/attributes:16794406/1099511627776/Minimum/vlen=15/mvcc=0
> followed by a smaller key //0/Minimum/vlen=0/mvcc=0 in cf attributes
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.checkScanOrder(StoreScanner.java:672)
> {noformat}
> It shows that the hfile seems to be corrupted -- the keys don't seem to be
> right.
> But Scanner is not able to give a meaningful error, but stuck in an infinite
> loop in here:
> {code}
> KeyValueHeap.generalizedSeek()
> while ((scanner = heap.poll()) != null) {
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511544#comment-15511544
]
Hadoop QA commented on HBASE-16672:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 38s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 35s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 35s {color} |
{color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 35s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 4s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 7s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 12s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 19s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 24s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 31s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 38s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 44s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 50s
{color} | {color:red} The patch causes 17 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511530#comment-15511530
]
Appy commented on HBASE-14734:
--
Patch isn't touching any of the failing tests. It touches HBaseTestingUtility,
but only adds a function without changing any existing logic. So the failures
are totally unrelated.
Still, ran the above timed out tests locally and they passed.
Committing.
{noformat}
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.client.TestHCM
^[[DTests run: 24, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 310.58 sec
- in org.apache.hadoop.hbase.client.TestHCM
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running
org.apache.hadoop.hbase.client.TestIncrementFromClientSideWithCoprocessor
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.301 sec - in
org.apache.hadoop.hbase.client.TestIncrementFromClientSideWithCoprocessor
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.521 sec -
in org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.client.TestMultiRespectsLimits
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.173 sec - in
org.apache.hadoop.hbase.client.TestMultiRespectsLimits
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.client.TestReplicasClient
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 254.478 sec -
in org.apache.hadoop.hbase.client.TestReplicasClient
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.408 sec - in
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.client.TestSnapshotMetadata
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.885 sec - in
org.apache.hadoop.hbase.client.TestSnapshotMetadata
Results :
Tests run: 61, Failures: 0, Errors: 0, Skipped: 1
{noformat}
> BindException when setting up MiniKdc
> -
>
> Key: HBASE-14734
> URL: https://issues.apache.org/jira/browse/HBASE-14734
> Project: HBase
> Issue Type: Sub-task
> Components: flakey, test
>Reporter: stack
>Assignee: Appy
> Attachments: HBASE-14734.master.001.patch
>
>
> Root cause : Port for kdc service gets selected in the constructor, but we
> bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() -->
> KdcServer.start(). In meantime, some other service can capture the port which
> results in BindException. The solution here is to catch the exception and
> retry.
> From
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/
> Error Message
> Address already in use
> Stacktrace
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
> at
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
> at
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
> at
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
> at
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
[jira] [Updated] (HBASE-16675) Average region size may be incorrect when there is region whose RegionLoad cannot be retrieved
[ https://issues.apache.org/jira/browse/HBASE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16675: --- Status: Patch Available (was: Open) > Average region size may be incorrect when there is region whose RegionLoad > cannot be retrieved > -- > > Key: HBASE-16675 > URL: https://issues.apache.org/jira/browse/HBASE-16675 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16675.v1.txt > > > HBASE-15933 fixed the NullPointerException bug. > When there is one or more region whose RegionLoad cannot be retrieved, the > average region size may be incorrect. > We should not use tableRegions.size() as denominator - the number of regions > whose RegionLoad can be retrieved should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16675) Average region size may be incorrect when there is region whose RegionLoad cannot be retrieved
[ https://issues.apache.org/jira/browse/HBASE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16675: --- Attachment: 16675.v1.txt > Average region size may be incorrect when there is region whose RegionLoad > cannot be retrieved > -- > > Key: HBASE-16675 > URL: https://issues.apache.org/jira/browse/HBASE-16675 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16675.v1.txt > > > HBASE-15933 fixed the NullPointerException bug. > When there is one or more region whose RegionLoad cannot be retrieved, the > average region size may be incorrect. > We should not use tableRegions.size() as denominator - the number of regions > whose RegionLoad can be retrieved should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16675) Average region size may be incorrect when there is region whose RegionLoad cannot be retrieved
Ted Yu created HBASE-16675: -- Summary: Average region size may be incorrect when there is region whose RegionLoad cannot be retrieved Key: HBASE-16675 URL: https://issues.apache.org/jira/browse/HBASE-16675 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu HBASE-15933 fixed the NullPointerException bug. When there is one or more region whose RegionLoad cannot be retrieved, the average region size may be incorrect. We should not use tableRegions.size() as denominator - the number of regions whose RegionLoad can be retrieved should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16674) fix flaky TestAsyncLogRolling.testLogRollOnDatanodeDeath()
Umesh Agashe created HBASE-16674:
Summary: fix flaky TestAsyncLogRolling.testLogRollOnDatanodeDeath()
Key: HBASE-16674
URL: https://issues.apache.org/jira/browse/HBASE-16674
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Umesh Agashe
Assignee: Umesh Agashe
This test shows up on flaky dashboard here:
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
{code}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
org.apache.hadoop.hbase.regionserver.wal.TestAsyncLogRolling.testLogRollOnDatanodeDeath(TestAsyncLogRolling.java:63)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511478#comment-15511478
]
Hadoop QA commented on HBASE-16662:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 23s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 17s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 44s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence |
| | org.apache.hadoop.hbase.client.TestAdmin1 |
| | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| | org.apache.hadoop.hbase.filter.TestMultiRowRangeFilter |
| |
org.apache.hadoop.hbase.client.TestCloneSnapshotFromClientWithRegionReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829448/HBASE-16662-master.patch
|
| JIRA Issue | HBASE-16662 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 43e77399c75c
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Status: Patch Available (was: Open)
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Attachment: 16672.v3.txt
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Status: Open (was: Patch Available)
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Attachment: 16672.v2.txt
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Assignee: Ted Yu
Status: Patch Available (was: Open)
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16670) Make RpcServer#processRequest logic more robust
[
https://issues.apache.org/jira/browse/HBASE-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511412#comment-15511412
]
Hadoop QA commented on HBASE-16670:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 46s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestAdmin2 |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829672/HBASE-16670.patch |
| JIRA Issue | HBASE-16670 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f2be3ab6ab4b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 5568929 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3636/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3636/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3636/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Updated] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16673: -- Description: New command-line option: -set setName. Will retrieve backup history for a particular backup set (was: New command-line option: -bset setName. Will retrieve backup history for a particular backup set) > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > New command-line option: -set setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16673: -- Description: New command-line option: -bset setName. Will retrieve backup history for a particular backup set > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > New command-line option: -bset setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16673) Enhance history command: history per backup set
Vladimir Rodionov created HBASE-16673: - Summary: Enhance history command: history per backup set Key: HBASE-16673 URL: https://issues.apache.org/jira/browse/HBASE-16673 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Attachment: HBASE-16656-v1.patch Patch v1. cc:[~tedyu] > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16656-v1.patch > > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Description: Default backup set name is "backup". If we do backup for a backup set "SomeSetName", by default, backup id will be generated in a form: *SomeSetName_timestamp*. The goal is to separate backup images between different sets. The history command will be updated and the new command - merge will use this backup names (prefixes) was: Default backup set name is "backup". If we do backup for a backup set "SomeSetName", by default, backup id will be generated in a form: *SomeSetName_timestamp*. User can override default behavior by providing backup prefix name in a command - line: *-backup_prefix * The goal is to separate backup images between different sets and backup destinations. The history command will be updated and the new command - merge will use this backup names (prefixes) > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Summary: BackupID must include backup set name (was: BackupID must include backup set name (by default) or can be set by user ) > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > User can override default behavior by providing backup prefix name in a > command - line: > *-backup_prefix * > The goal is to separate backup images between different sets and backup > destinations. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511381#comment-15511381
]
Hadoop QA commented on HBASE-14734:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 39s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestMultiRespectsLimits |
| | org.apache.hadoop.hbase.client.TestIncrementFromClientSideWithCoprocessor
|
| | org.apache.hadoop.hbase.client.TestSnapshotMetadata |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829654/HBASE-14734.master.001.patch
|
| JIRA Issue | HBASE-14734 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d24401a28685 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5568929 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3634/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3634/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Attachment: 16672.v1.txt
Draft patch.
Still needs to add client side config.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
> Attachments: 16672.v1.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16669) fix flaky TestAdmin#testmergeRegions
[
https://issues.apache.org/jira/browse/HBASE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511309#comment-15511309
]
Hadoop QA commented on HBASE-16669:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 20s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 24s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829659/hbase-16669.v1.patch |
| JIRA Issue | HBASE-16669 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 65b5b671fdf5 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5568929 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3635/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3635/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3635/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3635/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Updated] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16294:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
Ted Yu created HBASE-16672:
--
Summary: Add option for bulk load to copy hfile(s) instead of
renaming
Key: HBASE-16672
URL: https://issues.apache.org/jira/browse/HBASE-16672
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
While working on HBASE-14417, I found that taking full backup of table which
received bulk loaded hfiles and later gone through incremental restore failed
with the following exception:
{code}
2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
subprocedure-pool7-thread-1]
snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
Exception in SnapshotSubprocedurePool
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
does not exist:
hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
{code}
The cause was that the bulk loaded hfiles were renamed during bulk load step of
incremental restore.
To support incrementally restoring to multiple destinations, this issue adds
option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Description:
While working on HBASE-14417, I found that taking full backup of table which
received bulk loaded hfiles and later gone through incremental restore failed
with the following exception:
{code}
2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
subprocedure-pool7-thread-1]
snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
Exception in SnapshotSubprocedurePool
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
does not exist:
hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
{code}
The cause was that the bulk loaded hfiles in source table were renamed during
bulk load step of incremental restore.
To support incrementally restoring to multiple destinations, this issue adds
option which would always copy hfile(s) during bulk load.
was:
While working on HBASE-14417, I found that taking full backup of table which
received bulk loaded hfiles and later gone through incremental restore failed
with the following exception:
{code}
2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
subprocedure-pool7-thread-1]
snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
Exception in SnapshotSubprocedurePool
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
does not exist:
hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
{code}
The cause was that the bulk loaded hfiles were renamed during bulk load step of
incremental restore.
To support incrementally restoring to multiple destinations, this issue adds
option which would always copy hfile(s) during bulk load.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load
[jira] [Updated] (HBASE-16671) Split TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16671: Status: Patch Available (was: Open) > Split TestExportSnapshot > > > Key: HBASE-16671 > URL: https://issues.apache.org/jira/browse/HBASE-16671 > Project: HBase > Issue Type: Test > Components: snapshots, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16671-v0.patch > > > TestExportSnapshot contains 3 type of tests. > - MiniCluster creating a table, taking a snapshot and running export > - Mocked snapshot running export > - tool helpers tests > since now we have everything packed in a single test. 2 and 3 ended up having > a before and after that is creating a table and taking a snapshot which is > not used. Move those tests out and cut some time from the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16671) Split TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16671: Attachment: HBASE-16671-v0.patch > Split TestExportSnapshot > > > Key: HBASE-16671 > URL: https://issues.apache.org/jira/browse/HBASE-16671 > Project: HBase > Issue Type: Test > Components: snapshots, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16671-v0.patch > > > TestExportSnapshot contains 3 type of tests. > - MiniCluster creating a table, taking a snapshot and running export > - Mocked snapshot running export > - tool helpers tests > since now we have everything packed in a single test. 2 and 3 ended up having > a before and after that is creating a table and taking a snapshot which is > not used. Move those tests out and cut some time from the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16671) Split TestExportSnapshot
Matteo Bertozzi created HBASE-16671: --- Summary: Split TestExportSnapshot Key: HBASE-16671 URL: https://issues.apache.org/jira/browse/HBASE-16671 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0 Attachments: HBASE-16671-v0.patch TestExportSnapshot contains 3 type of tests. - MiniCluster creating a table, taking a snapshot and running export - Mocked snapshot running export - tool helpers tests since now we have everything packed in a single test. 2 and 3 ended up having a before and after that is creating a table and taking a snapshot which is not used. Move those tests out and cut some time from the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16669) fix flaky TestAdmin#testmergeRegions
[
https://issues.apache.org/jira/browse/HBASE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511238#comment-15511238
]
Appy commented on HBASE-16669:
--
+1.
> fix flaky TestAdmin#testmergeRegions
>
>
> Key: HBASE-16669
> URL: https://issues.apache.org/jira/browse/HBASE-16669
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16669.v1.patch
>
>
> Recent test runs show that this test is failing with these error messages:
> {code}
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1385)
> {code}
> or
> {code}
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1394)
> {code}
> looking at the code this indicates that merge operation did not complete or
> didn't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511134#comment-15511134
]
Appy commented on HBASE-14734:
--
Gave a try at test after the chat, but now it seems that we actually can't test
it. The problem is, we are dealing with ports over which we don't have full
control. We can block a port and make sure that kdc setup fails, but we can't
ensure an always available dedicated port to test that kdc setup succeeds.
Here's the test, not committing it but putting here for posterity.
{noformat}
@Test
void TestMiniKdcSetupBindException() throws Exception {
HBaseTestingUtility hbt = new HBaseTestingUtility();
ServerSocket socket = new ServerSocket(0);
int port = socket.getLocalPort();
boolean exception = false;
try {
hbt.setupMiniKdc(null, port);
} catch (BindException e) {
exception = true;
}
assertTrue("setupMiniKdc didn't encounter exception even though port was
not available.",
exception);
// Close the socket and try again. Shouldn't get exception now.
socket.close();
exception = false;
try {
hbt.setupMiniKdc(null, port);
} catch (BindException e) {
exception = true;
}
// It's possible that port gets captured by some other service. So this
test itself is flaky by
// nature.
assertTrue("setupMiniKdc encountered exception even though port was
closed.", exception);
}
{noformat}
Chatted with stack, will commit current patch itself.
> BindException when setting up MiniKdc
> -
>
> Key: HBASE-14734
> URL: https://issues.apache.org/jira/browse/HBASE-14734
> Project: HBase
> Issue Type: Sub-task
> Components: flakey, test
>Reporter: stack
>Assignee: Appy
> Attachments: HBASE-14734.master.001.patch
>
>
> Root cause : Port for kdc service gets selected in the constructor, but we
> bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() -->
> KdcServer.start(). In meantime, some other service can capture the port which
> results in BindException. The solution here is to catch the exception and
> retry.
> From
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/
> Error Message
> Address already in use
> Stacktrace
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
> at
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
> at
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
> at
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
> at
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Can this utility be made to not fail if address taken? Try another?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Lau updated HBASE-16662: Status: Patch Available (was: Open) Formally submit patch file (not sure if I did this right). > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12949) Scanner can be stuck in infinite loop if the HFile is corrupted
[
https://issues.apache.org/jira/browse/HBASE-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511105#comment-15511105
]
Jerry He commented on HBASE-12949:
--
I committed v3 patch (after rebase) finally. Thanks for the review [~stack],
[~tedyu] (a while back).
> Scanner can be stuck in infinite loop if the HFile is corrupted
> ---
>
> Key: HBASE-12949
> URL: https://issues.apache.org/jira/browse/HBASE-12949
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3, 0.98.10
>Reporter: Jerry He
> Attachments: HBASE-12949-branch-1-v3.patch, HBASE-12949-master-v2
> (1).patch, HBASE-12949-master-v2.patch, HBASE-12949-master-v2.patch,
> HBASE-12949-master-v2.patch, HBASE-12949-master-v3.patch,
> HBASE-12949-master.patch
>
>
> We've encountered problem where compaction hangs and never completes.
> After looking into it further, we found that the compaction scanner was stuck
> in a infinite loop. See stack below.
> {noformat}
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:296)
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257)
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697)
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:672)
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223)
> {noformat}
> We identified the hfile that seems to be corrupted. Using HFile tool shows
> the following:
> {noformat}
> [biadmin@hdtest009 bin]$ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -k
> -m -f
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> 15/01/23 11:53:17 INFO Configuration.deprecation: hadoop.native.lib is
> deprecated. Instead, use io.native.lib.available
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> 15/01/23 11:53:18 INFO Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> Scanning ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> WARNING, previous row is greater then current row
> filename ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> previous ->
> \x00/20110203-094231205-79442793-1410161293068203000\x0Aattributes16794406\x00\x00\x01\x00\x00\x00\x00\x00\x00
> current ->
> Exception in thread "main" java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:489)
> at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:347)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:856)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:768)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:362)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:262)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:220)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:539)
> at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:802)
> {noformat}
> Turning on Java Assert shows the following:
> {noformat}
> Exception in thread "main" java.lang.AssertionError: Key
> 20110203-094231205-79442793-1410161293068203000/attributes:16794406/1099511627776/Minimum/vlen=15/mvcc=0
> followed by a smaller key //0/Minimum/vlen=0/mvcc=0 in cf attributes
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.checkScanOrder(StoreScanner.java:672)
> {noformat}
> It shows that the hfile seems to be corrupted -- the keys don't seem to be
> right.
> But Scanner is not able to give a meaningful error, but stuck in an infinite
> loop in here:
> {code}
> KeyValueHeap.generalizedSeek()
> while ((scanner = heap.poll()) != null) {
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12949) Scanner can be stuck in infinite loop if the HFile is corrupted
[
https://issues.apache.org/jira/browse/HBASE-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-12949:
-
Resolution: Fixed
Assignee: Jerry He
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
2.0.0
Status: Resolved (was: Patch Available)
> Scanner can be stuck in infinite loop if the HFile is corrupted
> ---
>
> Key: HBASE-12949
> URL: https://issues.apache.org/jira/browse/HBASE-12949
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3, 0.98.10
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-12949-branch-1-v3.patch, HBASE-12949-master-v2
> (1).patch, HBASE-12949-master-v2.patch, HBASE-12949-master-v2.patch,
> HBASE-12949-master-v2.patch, HBASE-12949-master-v3.patch,
> HBASE-12949-master.patch
>
>
> We've encountered problem where compaction hangs and never completes.
> After looking into it further, we found that the compaction scanner was stuck
> in a infinite loop. See stack below.
> {noformat}
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:296)
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257)
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697)
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:672)
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223)
> {noformat}
> We identified the hfile that seems to be corrupted. Using HFile tool shows
> the following:
> {noformat}
> [biadmin@hdtest009 bin]$ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -k
> -m -f
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> 15/01/23 11:53:17 INFO Configuration.deprecation: hadoop.native.lib is
> deprecated. Instead, use io.native.lib.available
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> 15/01/23 11:53:18 INFO Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> Scanning ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> WARNING, previous row is greater then current row
> filename ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> previous ->
> \x00/20110203-094231205-79442793-1410161293068203000\x0Aattributes16794406\x00\x00\x01\x00\x00\x00\x00\x00\x00
> current ->
> Exception in thread "main" java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:489)
> at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:347)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:856)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:768)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:362)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:262)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:220)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:539)
> at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:802)
> {noformat}
> Turning on Java Assert shows the following:
> {noformat}
> Exception in thread "main" java.lang.AssertionError: Key
> 20110203-094231205-79442793-1410161293068203000/attributes:16794406/1099511627776/Minimum/vlen=15/mvcc=0
> followed by a smaller key //0/Minimum/vlen=0/mvcc=0 in cf attributes
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.checkScanOrder(StoreScanner.java:672)
> {noformat}
> It shows that the hfile seems to be corrupted -- the keys don't seem to be
> right.
> But Scanner is not able to give a meaningful error, but stuck in an infinite
> loop in here:
> {code}
> KeyValueHeap.generalizedSeek()
> while ((scanner = heap.poll()) != null) {
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16654) Better handle channelInactive and close for netty rpc client
[ https://issues.apache.org/jira/browse/HBASE-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511040#comment-15511040 ] Hudson commented on HBASE-16654: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1648 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1648/]) HBASE-16654 Better handle channelInactive and close for netty rpc client (zhangduo: rev 5568929dd2bfc20c1aa15d37d318459888cbd32a) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcDuplexHandler.java * (delete) hbase-client/src/main/java/org/apache/hadoop/hbase/security/AsyncHBaseSaslRpcClientHandler.java * (delete) hbase-client/src/main/java/org/apache/hadoop/hbase/security/AsyncHBaseSaslRpcClient.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BufferCallBeforeInitHandler.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslWrapHandler.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcConnection.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/security/NettyHBaseSaslRpcClientHandler.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/security/NettyHBaseSaslRpcClient.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslUnwrapHandler.java > Better handle channelInactive and close for netty rpc client > > > Key: HBASE-16654 > URL: https://issues.apache.org/jira/browse/HBASE-16654 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16654.patch > > > We should pass the event to the next handler in the pipeline as > channelInactive and close are usually used as the cleanup method of a > ChannelHandler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
