[jira] [Commented] (HBASE-16408) Handle On heap BucketCache size when HeapMemoryManager tunes memory
[ https://issues.apache.org/jira/browse/HBASE-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495603#comment-15495603 ] Anoop Sam John commented on HBASE-16408: As of now there is no problem (I was wrong).. Even if BucketCache is onheap, we dont consider that size at all in size calc for HeapMemory tuner.Now we have optimized off heap read so am wondering why we need on heap version of bucket cache. File and off heap mode is enough? Do some one really using BucketCache on heap? Off heap will perform equally or even better now. [~stack] how abt removing on heap Bucket cache mode? > Handle On heap BucketCache size when HeapMemoryManager tunes memory > --- > > Key: HBASE-16408 > URL: https://issues.apache.org/jira/browse/HBASE-16408 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16636) Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting Regions Stuck in Transition
[ https://issues.apache.org/jira/browse/HBASE-16636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495649#comment-15495649 ] Hari Sekhon commented on HBASE-16636: - Hi [~huaxiang], it is related although I've noticed that both the regions in transition as well as the regions in transition over threshold counts are both zero, are both fixed by that other issue? > Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting > Regions Stuck in Transition > - > > Key: HBASE-16636 > URL: https://issues.apache.org/jira/browse/HBASE-16636 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 1.1.2 > Environment: HDP 2.3.2 >Reporter: Hari Sekhon >Priority: Minor > Attachments: Regions_in_Transition_UI.png, ritCountOverThreshold.png > > > I've discovered that the Region in Transition counts are wrong in the HMaster > UI /jmx page. > The /master-status page clearly shows 3 regions stuck in transition but the > /jmx page I was monitoring reported 0 for ritCountOverThreshold. > {code} > }, { > "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger", > "modelerType" : "Master,sub=AssignmentManger", > "tag.Context" : "master", > ... > "ritOldestAge" : 0, > "ritCountOverThreshold" : 0, > ... > "ritCount" : 0, > {code} > I have a nagios plugin I wrote which was checking this which I've since had > to rewrite to parse the /master-status page instead (the code is in > check_hbase_regions_stuck_in_transition.py at > https://github.com/harisekhon/nagios-plugins). > I'm attaching screenshots of both /master-status and /jmx to show the > difference in the 2 pages on the HMaster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495675#comment-15495675 ] ramkrishna.s.vasudevan commented on HBASE-15871: I can change it. Thanks Ted for the review. Tried adding a 'closed' flag in SegmentScanner and to see how things work and found an interesting bug. So if we need to go with 'closed' flag in SegmentScanner and check for the flag on every operation then we need to solve the new bug. It is quite critical too. > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15871-branch-1.patch, > HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, > HBASE-15871.branch-1.1.003.patch, HBASE-15871.patch, HBASE-15871_1.patch, > HBASE-15871_1.patch, HBASE-15871_2.patch, HBASE-15871_3.patch, > HBASE-15871_4.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish > memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls > StoreScanner.updateReaders(), waits for acquiring a lock that store scanner > holds in StoreScanner.next() and backwardseek() in memstore scanner runs for > a long time. > I think that this condition could occur in reverse scan by the following > process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store > because all scanners was already closed by 2)'s flush() and backwardseek() > with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore > have sequenceID greater than this scanner's readPoint because of 2)'s > flush(). this condition causes searching all cells in memstore, and > seekToPreviousRow() repeatly seach cells that are already searched if a row > has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process > finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, > HBASE-14497's fix can't solve this issue because that fix just changed > recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to inproper ref counting of segments
ramkrishna.s.vasudevan created HBASE-16643: -- Summary: Reverse scanner heap creation may not allow MSLAB closure due to inproper ref counting of segments Key: HBASE-16643 URL: https://issues.apache.org/jira/browse/HBASE-16643 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical In the reverse scanner case, While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the backward heap, we do a {code} if ((backwardHeap == null) && (forwardHeap != null)) { forwardHeap.close(); forwardHeap = null; // before building the heap seek for the relevant key on the scanners, // for the heap to be built from the scanners correctly for (KeyValueScanner scan : scanners) { if (toLast) { res |= scan.seekToLastRow(); } else { res |= scan.backwardSeek(cell); } } {code} forwardHeap.close(). This would internally decrement the MSLAB ref counter for the current active segment and snapshot segment. When the scan is actually closed again we do close() and that will again decrement the count. Here chances are there that the count would go negative and hence the actual MSLAB closure that checks for refCount==0 will fail. Apart from this, when the refCount becomes 0 after the firstClose if any other thread requests to close the segment, then we will end up in corrupted segment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495687#comment-15495687 ] ramkrishna.s.vasudevan commented on HBASE-15871: I think for this JIRA we can commit it as is and then later in https://issues.apache.org/jira/browse/HBASE-16643 we can handle the close() properly. Without which adding a 'closed' flag is going to be logically wrong because once we set closed flag it cannot be again used. So we need to clearly check and fix that part. > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15871-branch-1.patch, > HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, > HBASE-15871.branch-1.1.003.patch, HBASE-15871.patch, HBASE-15871_1.patch, > HBASE-15871_1.patch, HBASE-15871_2.patch, HBASE-15871_3.patch, > HBASE-15871_4.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish > memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls > StoreScanner.updateReaders(), waits for acquiring a lock that store scanner > holds in StoreScanner.next() and backwardseek() in memstore scanner runs for > a long time. > I think that this condition could occur in reverse scan by the following > process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store > because all scanners was already closed by 2)'s flush() and backwardseek() > with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore > have sequenceID greater than this scanner's readPoint because of 2)'s > flush(). this condition causes searching all cells in memstore, and > seekToPreviousRow() repeatly seach cells that are already searched if a row > has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process > finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, > HBASE-14497's fix can't solve this issue because that fix just changed > recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to inproper ref counting of segments
[ https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16643: --- Description: In the reverse scanner case, While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the backward heap, we do a {code} if ((backwardHeap == null) && (forwardHeap != null)) { forwardHeap.close(); forwardHeap = null; // before building the heap seek for the relevant key on the scanners, // for the heap to be built from the scanners correctly for (KeyValueScanner scan : scanners) { if (toLast) { res |= scan.seekToLastRow(); } else { res |= scan.backwardSeek(cell); } } {code} forwardHeap.close(). This would internally decrement the MSLAB ref counter for the current active segment and snapshot segment. When the scan is actually closed again we do close() and that will again decrement the count. Here chances are there that the count would go negative and hence the actual MSLAB closure that checks for refCount==0 will fail. Apart from this, when the refCount becomes 0 after the firstClose if any other thread requests to close the segment, then we will end up in corrupted segment because the segment could be put back to the MSLAB pool. was: In the reverse scanner case, While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the backward heap, we do a {code} if ((backwardHeap == null) && (forwardHeap != null)) { forwardHeap.close(); forwardHeap = null; // before building the heap seek for the relevant key on the scanners, // for the heap to be built from the scanners correctly for (KeyValueScanner scan : scanners) { if (toLast) { res |= scan.seekToLastRow(); } else { res |= scan.backwardSeek(cell); } } {code} forwardHeap.close(). This would internally decrement the MSLAB ref counter for the current active segment and snapshot segment. When the scan is actually closed again we do close() and that will again decrement the count. Here chances are there that the count would go negative and hence the actual MSLAB closure that checks for refCount==0 will fail. Apart from this, when the refCount becomes 0 after the firstClose if any other thread requests to close the segment, then we will end up in corrupted segment. > Reverse scanner heap creation may not allow MSLAB closure due to inproper ref > counting of segments > -- > > Key: HBASE-16643 > URL: https://issues.apache.org/jira/browse/HBASE-16643 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > > In the reverse scanner case, > While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the > backward heap, we do a > {code} > if ((backwardHeap == null) && (forwardHeap != null)) { > forwardHeap.close(); > forwardHeap = null; > // before building the heap seek for the relevant key on the scanners, > // for the heap to be built from the scanners correctly > for (KeyValueScanner scan : scanners) { > if (toLast) { > res |= scan.seekToLastRow(); > } else { > res |= scan.backwardSeek(cell); > } > } > {code} > forwardHeap.close(). This would internally decrement the MSLAB ref counter > for the current active segment and snapshot segment. > When the scan is actually closed again we do close() and that will again > decrement the count. Here chances are there that the count would go negative > and hence the actual MSLAB closure that checks for refCount==0 will fail. > Apart from this, when the refCount becomes 0 after the firstClose if any > other thread requests to close the segment, then we will end up in corrupted > segment because the segment could be put back to the MSLAB pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf
[ https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495734#comment-15495734 ] ramkrishna.s.vasudevan commented on HBASE-16635: Thanks [~Apache9] Yes in the above condition it is the admin connection that is actually closed() but the Bytebuf created for that connection is not released. And reading the code, you are right we cannot release the Bytebuf in shutdown() because there the connection is put back to the pool for reusing it. > RpcClient under heavy load leaks some netty bytebuf > --- > > Key: HBASE-16635 > URL: https://issues.apache.org/jira/browse/HBASE-16635 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > > Yet to analyse the actual root cause. > But the case is that when we run a PE tool with 50 threads under heavy load > when the writes are clogged I think we have some netty Bytebuf leak. Not sure > if it is a serious issue but we get this log > {code} > 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16] > util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's > garbage-collected. Enable advanced leak reporting to find out where the leak > occurred. To enable advanced leak reporting, specify the JVM option > '-Dio.netty.leakDetection.level=advanced' or call > ResourceLeakDetector.setLevel() See > http://netty.io/wiki/reference-counted-objects.html for more information. > {code} > So reading the given link it is because of some ByteBuf that was not released > properly by the client and hence it gets GCed automatically. Netty provides > tips and tricks to find the root cause. Will get back here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16586) Procedure v2 - Cleanup sched wait/lock semantic
[ https://issues.apache.org/jira/browse/HBASE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495781#comment-15495781 ] Hadoop QA commented on HBASE-16586: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 31s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 49s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 26m 47s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 51s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 4s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 123m 4s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.security.token.TestGenerateDelegationToken | | Timed out junit tests | org.apache.hadoop.hbase.constraint.TestConstraint | | | org.apache.hadoop.hbase.TestNamespace | | | org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes | | | org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelReplicationWithExpAsString | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828799/HBASE-16586-v3.patch | | JIRA Issue | HBASE-16586 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 56d173564743 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / e19632a | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/3571/artifact/patchprocess/patch-unit-hbase-server.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/3571/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/3571/testReport/ | | modules | C: hbase-server U: hbase-server | | Console output | https://build
[jira] [Created] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3
Mikhail Antonov created HBASE-16644: --- Summary: Errors when reading legit HFile' Trailer on branch 1.3 Key: HBASE-16644 URL: https://issues.apache.org/jira/browse/HBASE-16644 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 1.3.0, 1.4.0 Reporter: Mikhail Antonov Assignee: Mikhail Antonov Fix For: 1.3.0 There seems to be a regression in branch 1.3 where we can't read HFile trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on some HFiles that could be successfully read on 1.2. I've seen this error manifesting in two ways so far. {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer from file at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) at org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) ... 6 more Caused by: java.io.IOException: Invalid HFile block magic: \x00\x00\x04\x00\x00\x00\x00\x00 at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155) at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167) at org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156) at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485) {code} and second {code} Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer from file at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) at org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) at org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) ... 6 more Caused by: java.io.IOException: Premature EOF from inputStream (read returned -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully read 1072 at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156) at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485) {code} In my case this problem was reproducible by
[jira] [Updated] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3
[ https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16644: Priority: Critical (was: Major) > Errors when reading legit HFile' Trailer on branch 1.3 > -- > > Key: HBASE-16644 > URL: https://issues.apache.org/jira/browse/HBASE-16644 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > > There seems to be a regression in branch 1.3 where we can't read HFile > trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on > some HFiles that could be successfully read on 1.2. > I've seen this error manifesting in two ways so far. > {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: > Problem reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Invalid HFile block magic: > \x00\x00\x04\x00\x00\x00\x00\x00 > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485) > {code} > and second > {code} > Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem > reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Premature EOF from inputStream (read returned > -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully > read 1072 > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397) > a
[jira] [Updated] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3
[ https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16644: Attachment: HBASE-16644.branch-1.3.patch > Errors when reading legit HFile' Trailer on branch 1.3 > -- > > Key: HBASE-16644 > URL: https://issues.apache.org/jira/browse/HBASE-16644 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-16644.branch-1.3.patch > > > There seems to be a regression in branch 1.3 where we can't read HFile > trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on > some HFiles that could be successfully read on 1.2. > I've seen this error manifesting in two ways so far. > {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: > Problem reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Invalid HFile block magic: > \x00\x00\x04\x00\x00\x00\x00\x00 > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485) > {code} > and second > {code} > Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem > reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Premature EOF from inputStream (read returned > -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully > read 1072 > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) > at > org.apache.hadoop.hbase.io.hfile.HFileBloc
[jira] [Updated] (HBASE-15134) Add visibility into Flush and Compaction queues
[ https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-15134: --- Attachment: HBASE-15134.master.001.patch > Add visibility into Flush and Compaction queues > --- > > Key: HBASE-15134 > URL: https://issues.apache.org/jira/browse/HBASE-15134 > Project: HBase > Issue Type: New Feature > Components: Compaction, metrics, regionserver >Reporter: Elliott Clark >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-15134.master.001.patch, HBASE-15134.patch, > HBASE-15134.patch > > > On busy spurts we can see regionservers start to see large queues for > compaction. It's really hard to tell if the server is queueing a lot of > compactions for the same region, lots of compactions for lots of regions, or > just falling behind. > For flushes much the same. There can be flushes in queue that aren't being > run because of delayed flushes. There's no way to know from the metrics how > many flushes are for each region, how many are delayed. Etc. > We should add either more metrics around this ( num per region, max per > region, min per region ) or add on a UI page that has the list of compactions > and flushes. > Or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15134) Add visibility into Flush and Compaction queues
[ https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496009#comment-15496009 ] Abhishek Singh Chouhan commented on HBASE-15134: Thanks [~busbey] will use the guidelines going forward :) Have added maxCompactionQueueSize and corresponding flush metric, which will get updated every 45 seconds as per the value at that point in time. Let me know if this looks fine. > Add visibility into Flush and Compaction queues > --- > > Key: HBASE-15134 > URL: https://issues.apache.org/jira/browse/HBASE-15134 > Project: HBase > Issue Type: New Feature > Components: Compaction, metrics, regionserver >Reporter: Elliott Clark >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-15134.master.001.patch, HBASE-15134.patch, > HBASE-15134.patch > > > On busy spurts we can see regionservers start to see large queues for > compaction. It's really hard to tell if the server is queueing a lot of > compactions for the same region, lots of compactions for lots of regions, or > just falling behind. > For flushes much the same. There can be flushes in queue that aren't being > run because of delayed flushes. There's no way to know from the metrics how > many flushes are for each region, how many are delayed. Etc. > We should add either more metrics around this ( num per region, max per > region, min per region ) or add on a UI page that has the list of compactions > and flushes. > Or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15134) Add visibility into Flush and Compaction queues
[ https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496256#comment-15496256 ] Hadoop QA commented on HBASE-15134: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 6s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 29s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 19s {color} | {color:red} hbase-hadoop2-compat in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 3s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 25m 58s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hbase-hadoop-compat in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s {color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 7s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 123m 18s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint | | | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures | | | org.apache.hadoop.hbase.master.TestTableLockManager | | | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828822/HBASE-15134.master.001.patch | | JIRA Issue | HBASE-15134 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 543c7d6b9061 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Perso
[jira] [Commented] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3
[ https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496255#comment-15496255 ] stack commented on HBASE-16644: --- Ouch. That looks like an ugly one to debug [~mantonov]. Sorry. Yeah, made a mistake presuming we always checksum (how did you write files w/o a checksum? A mistake?). Patch is great but needs a unit test, don't you think? > Errors when reading legit HFile' Trailer on branch 1.3 > -- > > Key: HBASE-16644 > URL: https://issues.apache.org/jira/browse/HBASE-16644 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-16644.branch-1.3.patch > > > There seems to be a regression in branch 1.3 where we can't read HFile > trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on > some HFiles that could be successfully read on 1.2. > I've seen this error manifesting in two ways so far. > {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: > Problem reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Invalid HFile block magic: > \x00\x00\x04\x00\x00\x00\x00\x00 > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485) > {code} > and second > {code} > Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem > reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Premature EOF from inputStream (read returned > -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully > read 1072 > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSR
[jira] [Commented] (HBASE-16408) Handle On heap BucketCache size when HeapMemoryManager tunes memory
[ https://issues.apache.org/jira/browse/HBASE-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496270#comment-15496270 ] stack commented on HBASE-16408: --- Will an on-heap bucketcache that does not make copies be faster than our lrublockcache? (Seems like it would be?) I'd say we can forbid it though to keep things simple (and given we don't have much practice doing onheap bucket cache). mmap'd is considered file-based? Its not looked on as onheap? > Handle On heap BucketCache size when HeapMemoryManager tunes memory > --- > > Key: HBASE-16408 > URL: https://issues.apache.org/jira/browse/HBASE-16408 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16631: -- Attachment: HBASE-16631.v2.patch Get another run in to see if the failures are related at all. > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, > HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15134) Add visibility into Flush and Compaction queues
[ https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-15134: --- Attachment: HBASE-15134.master.002.patch Fixing whitespace errors and getting another QA > Add visibility into Flush and Compaction queues > --- > > Key: HBASE-15134 > URL: https://issues.apache.org/jira/browse/HBASE-15134 > Project: HBase > Issue Type: New Feature > Components: Compaction, metrics, regionserver >Reporter: Elliott Clark >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-15134.master.001.patch, > HBASE-15134.master.002.patch, HBASE-15134.patch, HBASE-15134.patch > > > On busy spurts we can see regionservers start to see large queues for > compaction. It's really hard to tell if the server is queueing a lot of > compactions for the same region, lots of compactions for lots of regions, or > just falling behind. > For flushes much the same. There can be flushes in queue that aren't being > run because of delayed flushes. There's no way to know from the metrics how > many flushes are for each region, how many are delayed. Etc. > We should add either more metrics around this ( num per region, max per > region, min per region ) or add on a UI page that has the list of compactions > and flushes. > Or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496288#comment-15496288 ] stack commented on HBASE-16631: --- +1 on v1 (minus the pom change). Yeah, good idea making a new issue. I started another run just so can see if failures are related at all (they should not be given you made no change). > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, > HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496544#comment-15496544 ] Hadoop QA commented on HBASE-16631: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 54s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 21s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 24m 41s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 0s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 26s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 119m 48s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure | | | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures | | | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828828/HBASE-16631.v2.patch | | JIRA Issue | HBASE-16631 | | Optional Tests | asflicense javac javadoc unit xml compile findbugs hadoopcheck hbaseanti checkstyle | | uname | Linux db1a18b996d5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 2597217 | | Default Java |
[jira] [Commented] (HBASE-15134) Add visibility into Flush and Compaction queues
[ https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496565#comment-15496565 ] Hadoop QA commented on HBASE-15134: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 8s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 33s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 21s {color} | {color:red} hbase-hadoop2-compat in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 26m 40s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 46s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s {color} | {color:green} hbase-hadoop-compat in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s {color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 31s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 133m 29s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.regionserver.TestHRegion | | Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide | | | org.apache.hadoop.hbase.client.TestScannerTimeout | | | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas | | | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient | | | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828829/HBASE-15134.master.002.patch | | JIRA Issue | HBASE-15134 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7535a27e61f8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x
[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496580#comment-15496580 ] Nick Dimiduk commented on HBASE-15523: -- This sounds like the realm of system management tools -- write an init or upstart or whatever script for it, right? I have one in https://github.com/ndimiduk/cesnet-hbase/blob/hdp/templates/services/hbase-regionserver.conf.erb Maybe whatever tools builds the RPM should also include system service scripts for its target platform. See also http://mail-archives.apache.org/mod_mbox/hbase-dev/201606.mbox/%3CCANZa=GsMsAz0R=BKzGO7ThZNHbh=HS5L=evofnb-a7_3bds...@mail.gmail.com%3E > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Attachments: HBASE-15523.patch > > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496591#comment-15496591 ] Dima Spivak commented on HBASE-15523: - Not sure why this was reopened. Autorestart works and it shouldn't be up to HBase to fix a problem in Ambari. > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Attachments: HBASE-15523.patch > > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16349) TestClusterId may hang during cluster shutdown
[ https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496619#comment-15496619 ] Hudson commented on HBASE-16349: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1613 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1613/]) HBASE-16349 TestClusterId may hang during cluster shutdown (tedyu: rev 2597217ae5aa057e1931c772139ce8cc7a2b3efb) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestClusterId.java > TestClusterId may hang during cluster shutdown > -- > > Key: HBASE-16349 > URL: https://issues.apache.org/jira/browse/HBASE-16349 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Priority: Minor > Attachments: 16349.branch-1.v1.txt, 16349.v1.txt > > > I was running TestClusterId on branch-1 where I observed the test hang during > test tearDown(). > {code} > 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(1415): Closing hbase:meta,,1.1588230740: disabling > compactions & flushes > 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(1442): Updates disabled for region > hbase:meta,,1.1588230740 > 2016-08-03 11:36:39,600 INFO [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(2253): Flushing 1/1 column families, memstore=232 B > 2016-08-03 11:36:39,601 WARN [RS_OPEN_META-cn012:49371-0.append-pool17-t1] > wal.FSHLog$RingBufferEventHandler(1900): Append sequenceId=8, requesting roll > of WAL > java.io.IOException: All datanodes > DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK] > are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402) > 2016-08-03 11:36:39,602 FATAL [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegionServer(2085): ABORTING region server > cn012.l42scl.hortonworks.com,49371,1470249187586: Unrecoverable > exception while closing region hbase:meta,,1.1588230740, still finishing close > org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append > sequenceId=8, requesting roll of WAL > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1902) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1754) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1676) > at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: All datanodes > DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK] > are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402) > 2016-08-03 11:36:39,603 FATAL [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegionServer(2093): RegionServer abort: loaded coprocessors > are: [org.apache.hadoop.hbase.coprocessor. MultiRowMutationEndpoint] > {code} > This led to rst.join() hanging: > {code} > "RS:0;cn012:49371" #648 prio=5 os_prio=0 tid=0x7fdab24b5000 nid=0x621a > waiting on condition [0x7fd785fe] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1326) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1312) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1082) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16586) Procedure v2 - Cleanup sched wait/lock semantic
[ https://issues.apache.org/jira/browse/HBASE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16586: Resolution: Fixed Status: Resolved (was: Patch Available) > Procedure v2 - Cleanup sched wait/lock semantic > --- > > Key: HBASE-16586 > URL: https://issues.apache.org/jira/browse/HBASE-16586 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16586-v0.patch, HBASE-16586-v1.patch, > HBASE-16586-v2.patch, HBASE-16586-v3.patch > > > For some reason waitEvent() and waitRegion() had a mismatching return value. > unity the wait semantic in being: return true we wait, false we don't wait. > procedures using hasLock = waitRegion() should change to hasLock = > !waitRegion(). at the moment we have only DispatchMergingRegionsProcedure > using it (in master). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16636) Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting Regions Stuck in Transition
[ https://issues.apache.org/jira/browse/HBASE-16636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496683#comment-15496683 ] huaxiang sun commented on HBASE-16636: -- [~harisekhon], it seems to be fixed by HBASE-14644, I did a similar test with the fix before. > Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting > Regions Stuck in Transition > - > > Key: HBASE-16636 > URL: https://issues.apache.org/jira/browse/HBASE-16636 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 1.1.2 > Environment: HDP 2.3.2 >Reporter: Hari Sekhon >Priority: Minor > Attachments: Regions_in_Transition_UI.png, ritCountOverThreshold.png > > > I've discovered that the Region in Transition counts are wrong in the HMaster > UI /jmx page. > The /master-status page clearly shows 3 regions stuck in transition but the > /jmx page I was monitoring reported 0 for ritCountOverThreshold. > {code} > }, { > "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger", > "modelerType" : "Master,sub=AssignmentManger", > "tag.Context" : "master", > ... > "ritOldestAge" : 0, > "ritCountOverThreshold" : 0, > ... > "ritCount" : 0, > {code} > I have a nagios plugin I wrote which was checking this which I've since had > to rewrite to parse the /master-status page instead (the code is in > check_hbase_regions_stuck_in_transition.py at > https://github.com/harisekhon/nagios-plugins). > I'm attaching screenshots of both /master-status and /jmx to show the > difference in the 2 pages on the HMaster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16598: - Attachment: (was: HBASE-16598-v4.patch) > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, > HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16598: - Attachment: HBASE-16598-v4.patch > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, > HBASE-16598-v4.patch, HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496759#comment-15496759 ] Jerry He commented on HBASE-15523: -- Yes. autorestart is supported in hbase-daemon.sh. hbase-daemon.sh start|stop|restart|autorestart I think the argument is that 'autorestart' is behavior option or extension for the operation 'start'. It is better and more flexible to control this behavior option via other means, instead of having a separate command. In the Ambari case, even it were going to be built into Ambari, for a user to be able to control and switch dynamically on the Ambari console, it probably still need a dynamic switch on the console, and either 'start' or 'autorestart' is invoked based on the switch. Then why not have this dynamic switch in HBase? We can introduce this env property, and deprecate the ''autorestart' command. My two cents. > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Attachments: HBASE-15523.patch > > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496770#comment-15496770 ] Heng Chen commented on HBASE-16631: --- Timeout test case could pass locally, relates testcase such as TestAsyncProcess, TestFromClientSideXXX has passed. will commit it to master > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, > HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16631: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, > HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496793#comment-15496793 ] Dima Spivak commented on HBASE-15523: - Moving this to a variable (that would ostensibly need to be set on every machine in a cluster) moves the onus from the script to the user and also means you couldn't grep your cluster's restart behavior via looking at the process tree. For those reasons, I'm not a fan of this proposal (and I'm still not clear on why it was reopened after being closed for six months). > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Attachments: HBASE-15523.patch > > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
ChiaPing Tsai created HBASE-16645: - Summary: Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap Key: HBASE-16645 URL: https://issues.apache.org/jira/browse/HBASE-16645 Project: HBase Issue Type: Bug Affects Versions: 2.0.0 Reporter: ChiaPing Tsai Priority: Minor Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
[ https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16645: -- Attachment: HBASE-16645.v0.patch > Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap > -- > > Key: HBASE-16645 > URL: https://issues.apache.org/jira/browse/HBASE-16645 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16645.v0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
[ https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16645: -- Status: Patch Available (was: Open) > Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap > -- > > Key: HBASE-16645 > URL: https://issues.apache.org/jira/browse/HBASE-16645 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16645.v0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
[ https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16645: -- Description: Two reasons are shown below: 1) CellFlatMap#find doesn’t consider desc order array 2) CellFlatMap#getValidIndex return the wrong upper bound > Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap > -- > > Key: HBASE-16645 > URL: https://issues.apache.org/jira/browse/HBASE-16645 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16645.v0.patch > > > Two reasons are shown below: > 1) CellFlatMap#find doesn’t consider desc order array > 2) CellFlatMap#getValidIndex return the wrong upper bound -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496861#comment-15496861 ] Yi Liang commented on HBASE-15523: -- I closed this one on Aug by accident, I reopen it to see if anyone have any other suggestion or idea of this proposal. From my point of view, the autostart command is also a kind of extension for start command, we need somehow extend start command to support this autostart, not have a new command for autostart. > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Attachments: HBASE-15523.patch > > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496891#comment-15496891 ] Nick Dimiduk commented on HBASE-15523: -- IMHO adding an optional param makes more sense than "hiding" the behavior behind an ENV variable. Would be much easier to diagnose/debug when shit hits the fan in the middle of the night when the feature is exposed right there in the launch command. > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Attachments: HBASE-15523.patch > > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496911#comment-15496911 ] Jerry He commented on HBASE-15523: -- ok. It is hard to make all happy :-( One wants to control it via hbase-env, and others wants to control via a visible command. > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Attachments: HBASE-15523.patch > > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16349) TestClusterId may hang during cluster shutdown
[ https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496909#comment-15496909 ] Hudson commented on HBASE-16349: FAILURE: Integrated in Jenkins build HBase-1.4 #418 (See [https://builds.apache.org/job/HBase-1.4/418/]) HBASE-16349 TestClusterId may hang during cluster shutdown (tedyu: rev 591cc4cfb8c908556eb6e64811aa7b46fb8bbb7a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestClusterId.java > TestClusterId may hang during cluster shutdown > -- > > Key: HBASE-16349 > URL: https://issues.apache.org/jira/browse/HBASE-16349 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Priority: Minor > Attachments: 16349.branch-1.v1.txt, 16349.v1.txt > > > I was running TestClusterId on branch-1 where I observed the test hang during > test tearDown(). > {code} > 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(1415): Closing hbase:meta,,1.1588230740: disabling > compactions & flushes > 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(1442): Updates disabled for region > hbase:meta,,1.1588230740 > 2016-08-03 11:36:39,600 INFO [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(2253): Flushing 1/1 column families, memstore=232 B > 2016-08-03 11:36:39,601 WARN [RS_OPEN_META-cn012:49371-0.append-pool17-t1] > wal.FSHLog$RingBufferEventHandler(1900): Append sequenceId=8, requesting roll > of WAL > java.io.IOException: All datanodes > DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK] > are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402) > 2016-08-03 11:36:39,602 FATAL [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegionServer(2085): ABORTING region server > cn012.l42scl.hortonworks.com,49371,1470249187586: Unrecoverable > exception while closing region hbase:meta,,1.1588230740, still finishing close > org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append > sequenceId=8, requesting roll of WAL > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1902) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1754) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1676) > at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: All datanodes > DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK] > are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402) > 2016-08-03 11:36:39,603 FATAL [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegionServer(2093): RegionServer abort: loaded coprocessors > are: [org.apache.hadoop.hbase.coprocessor. MultiRowMutationEndpoint] > {code} > This led to rst.join() hanging: > {code} > "RS:0;cn012:49371" #648 prio=5 os_prio=0 tid=0x7fdab24b5000 nid=0x621a > waiting on condition [0x7fd785fe] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1326) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1312) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1082) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496931#comment-15496931 ] Jerry He commented on HBASE-16598: -- All the failed/timeout tests passed locally (multiple times) {noformat} Running org.apache.hadoop.hbase.TestMultiVersions Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.537 sec - in org.apache.hadoop.hbase.TestMultiVersions Running org.apache.hadoop.hbase.security.visibility.TestWithDisabledAuthorization Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.159 sec - in org.apache.hadoop.hbase.security.visibility.TestWithDisabledAuthorization Running org.apache.hadoop.hbase.security.access.TestAccessController Tests run: 66, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.392 sec - in org.apache.hadoop.hbase.security.access.TestAccessController Running org.apache.hadoop.hbase.security.access.TestWithDisabledAuthorization Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.62 sec - in org.apache.hadoop.hbase.security.access.TestWithDisabledAuthorization Running org.apache.hadoop.hbase.security.access.TestNamespaceCommands Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.432 sec - in org.apache.hadoop.hbase.security.access.TestNamespaceCommands Running org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 272.686 sec - in org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot Running org.apache.hadoop.hbase.snapshot.TestExportSnapshot Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 273.527 sec - in org.apache.hadoop.hbase.snapshot.TestExportSnapshot Results : Tests run: 102, Failures: 0, Errors: 0, Skipped: 0 {noformat} > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, > HBASE-16598-v4.patch, HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496934#comment-15496934 ] Jerry He commented on HBASE-16598: -- [~mbertozzi] [~ashish singhi] Are you good with the patch v4? > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, > HBASE-16598-v4.patch, HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496953#comment-15496953 ] Matteo Bertozzi commented on HBASE-16598: - looks good to me, +1 > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, > HBASE-16598-v4.patch, HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide
[ https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-16544: - Attachment: HBASE-16544-addendum.patch forget to remove links to the removed section 'using amazon s3'. Provide addendum to remove it. > Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide > --- > > Key: HBASE-16544 > URL: https://issues.apache.org/jira/browse/HBASE-16544 > Project: HBase > Issue Type: Bug > Components: documentation, Filesystem Integration >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16544-V1.patch, HBASE-16544-addendum.patch > > > reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration > (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I > think this title means that we can use S3 storage to replace HDFS, I really > tried this :(, always give me errors and hbase even can not start, see > error mentioned in jira HBASE-11045. > (2) And the details in this section are more about deploying HBase on Amazon > EC2 cluster, which has nothing to do 'using Amazon S3 storage' > (3) In all, I think we need to remove this section, or at least clarify this > section if someone fully test HBase on S3. see HBASE-15646 for more details > about this doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide
[ https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497015#comment-15497015 ] Matteo Bertozzi commented on HBASE-16544: - +1 > Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide > --- > > Key: HBASE-16544 > URL: https://issues.apache.org/jira/browse/HBASE-16544 > Project: HBase > Issue Type: Bug > Components: documentation, Filesystem Integration >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16544-V1.patch, HBASE-16544-addendum.patch > > > reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration > (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I > think this title means that we can use S3 storage to replace HDFS, I really > tried this :(, always give me errors and hbase even can not start, see > error mentioned in jira HBASE-11045. > (2) And the details in this section are more about deploying HBase on Amazon > EC2 cluster, which has nothing to do 'using Amazon S3 storage' > (3) In all, I think we need to remove this section, or at least clarify this > section if someone fully test HBase on S3. see HBASE-15646 for more details > about this doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3
[ https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497029#comment-15497029 ] Mikhail Antonov commented on HBASE-16644: - [~stack] thanks for quick reply! So far my thoughts on it: - that seems to only happen under specific conditions, since I've only seen a handful of such cases. Still looking at what's special about those files. So far it *seems" that it only happens on specific blocks, the code path in HFileReaderV2 constructor when we read meta index (data index was read fine). - Debugging and reproducing is kind of painful as I've only seen that on real files and don't yet have cooked-up test file that can demonstrate this effect on. - Regardless of how such a file was written, still if we can read some file on 1.2 and we can't on 1.3 that is a critical regression on the read path IMO. > Errors when reading legit HFile' Trailer on branch 1.3 > -- > > Key: HBASE-16644 > URL: https://issues.apache.org/jira/browse/HBASE-16644 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-16644.branch-1.3.patch > > > There seems to be a regression in branch 1.3 where we can't read HFile > trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on > some HFiles that could be successfully read on 1.2. > I've seen this error manifesting in two ways so far. > {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: > Problem reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Invalid HFile block magic: > \x00\x00\x04\x00\x00\x00\x00\x00 > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485) > {code} > and second > {code} > Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem > reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HS
[jira] [Commented] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3
[ https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497039#comment-15497039 ] stack commented on HBASE-16644: --- bq. Regardless of how such a file was written, still if we can read some file on 1.2 and we can't on 1.3 that is a critical regression on the read path IMO. Agree. > Errors when reading legit HFile' Trailer on branch 1.3 > -- > > Key: HBASE-16644 > URL: https://issues.apache.org/jira/browse/HBASE-16644 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-16644.branch-1.3.patch > > > There seems to be a regression in branch 1.3 where we can't read HFile > trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on > some HFiles that could be successfully read on 1.2. > I've seen this error manifesting in two ways so far. > {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: > Problem reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Invalid HFile block magic: > \x00\x00\x04\x00\x00\x00\x00\x00 > at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155) > at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485) > {code} > and second > {code} > Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem > reading HFile Trailer from file > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164) > at > org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) > at > org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516) > ... 6 more > Caused by: java.io.IOException: Premature EOF from inputStream (read returned > -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully > read 1072 > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712) >
[jira] [Updated] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution
[ https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16587: Attachment: HBASE-16587-v1.patch > Procedure v2 - Cleanup suspended proc execution > --- > > Key: HBASE-16587 > URL: https://issues.apache.org/jira/browse/HBASE-16587 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16587-v0.patch, HBASE-16587-v1.patch > > > for procedures like the assignment or the lock one we need to be able to hold > on locks while suspended. At the moment the way to do that is up to the proc > implementation. This patch moves the logic to the base Procedure and > ProcedureExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution
[ https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16587: Status: Patch Available (was: Open) > Procedure v2 - Cleanup suspended proc execution > --- > > Key: HBASE-16587 > URL: https://issues.apache.org/jira/browse/HBASE-16587 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16587-v0.patch, HBASE-16587-v1.patch > > > for procedures like the assignment or the lock one we need to be able to hold > on locks while suspended. At the moment the way to do that is up to the proc > implementation. This patch moves the logic to the base Procedure and > ProcedureExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
[ https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497085#comment-15497085 ] Ted Yu commented on HBASE-16645: [~anastas]: Can you take a look ? > Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap > -- > > Key: HBASE-16645 > URL: https://issues.apache.org/jira/browse/HBASE-16645 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16645.v0.patch > > > Two reasons are shown below: > 1) CellFlatMap#find doesn’t consider desc order array > 2) CellFlatMap#getValidIndex return the wrong upper bound -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16447) Replication by namespaces config in peer
[ https://issues.apache.org/jira/browse/HBASE-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-16447: -- Fix Version/s: 1.4.0 2.0.0 > Replication by namespaces config in peer > > > Key: HBASE-16447 > URL: https://issues.apache.org/jira/browse/HBASE-16447 > Project: HBase > Issue Type: New Feature > Components: Replication >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16447-v1.patch, HBASE-16447-v2.patch, > HBASE-16447-v3.patch, HBASE-16447-v4.patch, HBASE-16447-v5.patch, > HBASE-16447-v5.patch > > > Now we only config table cfs in peer. But in our production cluster, there > are a dozen of namespace and every namespace has dozens of tables. It was > complicated to config all table cfs in peer. For some namespace, it need > replication all tables to other slave cluster. It will be easy to config if > we support replication by namespace. Suggestions and discussions are welcomed. > Review board: https://reviews.apache.org/r/51521/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16447) Replication by namespaces config in peer
[ https://issues.apache.org/jira/browse/HBASE-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497126#comment-15497126 ] Enis Soztutar commented on HBASE-16447: --- Thanks Guanghao Zhang for the updated patch. I've committed this to master. branch-1 has a lot of conflicts, do you mind attaching a patch for branch-1 as well. > Replication by namespaces config in peer > > > Key: HBASE-16447 > URL: https://issues.apache.org/jira/browse/HBASE-16447 > Project: HBase > Issue Type: New Feature > Components: Replication >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16447-v1.patch, HBASE-16447-v2.patch, > HBASE-16447-v3.patch, HBASE-16447-v4.patch, HBASE-16447-v5.patch, > HBASE-16447-v5.patch > > > Now we only config table cfs in peer. But in our production cluster, there > are a dozen of namespace and every namespace has dozens of tables. It was > complicated to config all table cfs in peer. For some namespace, it need > replication all tables to other slave cluster. It will be easy to config if > we support replication by namespace. Suggestions and discussions are welcomed. > Review board: https://reviews.apache.org/r/51521/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497205#comment-15497205 ] Hudson commented on HBASE-16631: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1614 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1614/]) HBASE-16631 Extract AsyncRequestFuture related code from AsyncProcess (chenheng: rev 2cf8907db53b84a0118acc1edd1dfb9b37abe8b7) * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BatchErrors.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFuture.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicasClient.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java > Extract AsyncRequestFuture related code from AsyncProcess > - > > Key: HBASE-16631 > URL: https://issues.apache.org/jira/browse/HBASE-16631 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen >Assignee: Heng Chen > Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, > HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch > > > Now, AsyncProcess class is too large (over 2000+ lines), and there are so > many sub classes in it. > AsyncRequestFutureImpl is the biggest subclass in AP, we could extract it > out from AP to reduce the AP size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16586) Procedure v2 - Cleanup sched wait/lock semantic
[ https://issues.apache.org/jira/browse/HBASE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497206#comment-15497206 ] Hudson commented on HBASE-16586: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1614 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1614/]) HBASE-16586 Procedure v2 - Cleanup sched wait/lock semantic (matteo.bertozzi: rev b6b72361b68634f15f8cf83738d89147633ac378) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureScheduler.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DispatchMergingRegionsProcedure.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java > Procedure v2 - Cleanup sched wait/lock semantic > --- > > Key: HBASE-16586 > URL: https://issues.apache.org/jira/browse/HBASE-16586 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16586-v0.patch, HBASE-16586-v1.patch, > HBASE-16586-v2.patch, HBASE-16586-v3.patch > > > For some reason waitEvent() and waitRegion() had a mismatching return value. > unity the wait semantic in being: return true we wait, false we don't wait. > procedures using hasLock = waitRegion() should change to hasLock = > !waitRegion(). at the moment we have only DispatchMergingRegionsProcedure > using it (in master). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
[ https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497219#comment-15497219 ] Hadoop QA commented on HBASE-16645: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 12s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 57s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 51s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 32m 15s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 116m 54s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 167m 6s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures | | Timed out junit tests | org.apache.hadoop.hbase.security.access.TestCellACLWithMultipleVersions | | | org.apache.hadoop.hbase.security.token.TestDelegationTokenWithEncryption | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.1 Server=1.12.1 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828855/HBASE-16645.v0.patch | | JIRA Issue | HBASE-16645 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5ea5176f7bf8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 2cf8907 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/3576/artifact/patchprocess/patch-unit-hbase-server.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/3576/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/3576/testReport/ | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/3576/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This mess
[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497321#comment-15497321 ] Enis Soztutar commented on HBASE-16598: --- Great cleanup. +1. > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, > HBASE-16598-v4.patch, HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution
[ https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497334#comment-15497334 ] Hadoop QA commented on HBASE-16587: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 49s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 22s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 24m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s {color} | {color:green} hbase-procedure in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 29s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 116m 1s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hbase.master.TestMasterFailover | | | org.apache.hadoop.hbase.master.TestTableLockManager | | | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache | | | org.apache.hadoop.hbase.io.hfile.TestScannerFromBucketCache | | | org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite | | | org.apache.hadoop.hbase.master.TestMetaShutdownHandler | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828883/HBASE-16587-v1.patch | | JIRA Issue | HBASE-16587 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux b26bf3a4ced4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497422#comment-15497422 ] Enis Soztutar commented on HBASE-16257: --- Thanks Jerry for the latest patch. Sorry, but one last item may need to be addressed. Otherwise, everything looks good. The HFileReplicator now uses the bulk load staging dir as a path under root. This will become a problem if the source cluster is upgraded first before the sink cluster. In case of cyclic replication, it will always be the case that you can have 2.x -> 1.x replication. If that is the case, the staging dir will not be there under the 1.x root, which will then case the HFileReplicator to fail. Maybe we can do a simple exists check on the remote FS to see whether we can use the new hbase.root.dir/staging or we should fall back to using the value from configuration. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input
Ted Yu created HBASE-16646: -- Summary: Enhance LoadIncrementalHFiles to accept store file paths as input Key: HBASE-16646 URL: https://issues.apache.org/jira/browse/HBASE-16646 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Currently LoadIncrementalHFiles takes the directory (output path) as input parameter. In some scenarios (incremental restore of bulk loaded hfiles), the List of paths to hfiles is known. LoadIncrementalHFiles can take the List as input parameter and proceed with loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Attachment: 16646.v0.txt Patch v0 illustrates the tentative change. List of hfiles may be retrieved from BulkLoadDescriptor. > Enhance LoadIncrementalHFiles to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497446#comment-15497446 ] Hadoop QA commented on HBASE-16598: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 25s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 5s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 18m 32s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 33s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patched modules with no Java source: . {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 37s {color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 45s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 18m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 1s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patched modules with no Java source: . {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s {color} | {color:green} hbase-protocol in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 39s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 6s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s {color} | {color:green} hbase-rsgroup in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 57s {color} | {color:green} hbase-shell in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 57s {color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 23s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color
[jira] [Created] (HBASE-16647) hbck should do the offline reference repair before online repair
Jerry He created HBASE-16647: Summary: hbck should do the offline reference repair before online repair Key: HBASE-16647 URL: https://issues.apache.org/jira/browse/HBASE-16647 Project: HBase Issue Type: Bug Reporter: Jerry He Assignee: Jerry He {noformat} hbck -fixReferenceFiles Try to offline lingering reference store files Metadata Repair shortcuts -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes {noformat} Bad reference files prevent the region from coming online. If used in the shortcut combination, the reference files should be fixed before other online fix. I have seen repeated '-repair' did not work because bad reference files failed regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16647) hbck should do offline reference repair before online repair
[ https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16647: - Summary: hbck should do offline reference repair before online repair (was: hbck should do the offline reference repair before online repair) > hbck should do offline reference repair before online repair > > > Key: HBASE-16647 > URL: https://issues.apache.org/jira/browse/HBASE-16647 > Project: HBase > Issue Type: Bug >Reporter: Jerry He >Assignee: Jerry He > > {noformat} > hbck > -fixReferenceFiles Try to offline lingering reference store files > Metadata Repair shortcuts > -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles > -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps > -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes > {noformat} > Bad reference files prevent the region from coming online. > If used in the shortcut combination, the reference files should be fixed > before other online fix. > I have seen repeated '-repair' did not work because bad reference files > failed regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Attachment: 16646.v1.txt In patch v1, signatures of public methods are kept. > Enhance LoadIncrementalHFiles to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Attachment: (was: 16646.v1.txt) > Enhance LoadIncrementalHFiles to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16447) Replication by namespaces config in peer
[ https://issues.apache.org/jira/browse/HBASE-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497671#comment-15497671 ] Hudson commented on HBASE-16447: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1615 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1615/]) HBASE-16447 Replication by namespaces config in peer (Guanghao Zhang) (enis: rev 1a1003a482d9bfb725fbe1097c794fdb043dcd81) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java * (add) hbase-shell/src/main/ruby/shell/commands/set_peer_namespaces.rb * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/NamespaceTableCfWALEntryFilter.java * (edit) hbase-shell/src/test/ruby/hbase/replication_admin_test.rb * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerZKImpl.java * (edit) hbase-shell/src/main/ruby/shell/commands/add_peer.rb * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java * (edit) hbase-shell/src/main/ruby/shell.rb * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java * (edit) hbase-shell/src/main/ruby/shell/commands/set_peer_tableCFs.rb * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerConfig.java * (edit) hbase-shell/src/main/ruby/hbase_constants.rb * (edit) hbase-protocol/src/main/protobuf/ZooKeeper.proto * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestProtos.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationSerDeHelper.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java * (edit) hbase-shell/src/main/ruby/hbase/replication_admin.rb * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ZKNamespaceManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java * (edit) hbase-shell/src/main/ruby/shell/commands/list_peers.rb * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestNamespaceReplication.java > Replication by namespaces config in peer > > > Key: HBASE-16447 > URL: https://issues.apache.org/jira/browse/HBASE-16447 > Project: HBase > Issue Type: New Feature > Components: Replication >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16447-v1.patch, HBASE-16447-v2.patch, > HBASE-16447-v3.patch, HBASE-16447-v4.patch, HBASE-16447-v5.patch, > HBASE-16447-v5.patch > > > Now we only config table cfs in peer. But in our production cluster, there > are a dozen of namespace and every namespace has dozens of tables. It was > complicated to config all table cfs in peer. For some namespace, it need > replication all tables to other slave cluster. It will be easy to config if > we support replication by namespace. Suggestions and discussions are welcomed. > Review board: https://reviews.apache.org/r/51521/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Attachment: 16646.v1.txt > Enhance LoadIncrementalHFiles to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497674#comment-15497674 ] Matteo Bertozzi commented on HBASE-16646: - how does it find which family the file belongs to? is the assumption here that you have the family name as parent dir? so something like LoadIncremental /myStoreFile does not work but something like LoadIncremental /myFamily/myStoreFile works? > Enhance LoadIncrementalHFiles to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16349) TestClusterId may hang during cluster shutdown
[ https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16349: --- Issue Type: Test (was: Bug) > TestClusterId may hang during cluster shutdown > -- > > Key: HBASE-16349 > URL: https://issues.apache.org/jira/browse/HBASE-16349 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: 16349.branch-1.v1.txt, 16349.v1.txt > > > I was running TestClusterId on branch-1 where I observed the test hang during > test tearDown(). > {code} > 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(1415): Closing hbase:meta,,1.1588230740: disabling > compactions & flushes > 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(1442): Updates disabled for region > hbase:meta,,1.1588230740 > 2016-08-03 11:36:39,600 INFO [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(2253): Flushing 1/1 column families, memstore=232 B > 2016-08-03 11:36:39,601 WARN [RS_OPEN_META-cn012:49371-0.append-pool17-t1] > wal.FSHLog$RingBufferEventHandler(1900): Append sequenceId=8, requesting roll > of WAL > java.io.IOException: All datanodes > DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK] > are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402) > 2016-08-03 11:36:39,602 FATAL [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegionServer(2085): ABORTING region server > cn012.l42scl.hortonworks.com,49371,1470249187586: Unrecoverable > exception while closing region hbase:meta,,1.1588230740, still finishing close > org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append > sequenceId=8, requesting roll of WAL > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1902) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1754) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1676) > at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: All datanodes > DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK] > are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402) > 2016-08-03 11:36:39,603 FATAL [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegionServer(2093): RegionServer abort: loaded coprocessors > are: [org.apache.hadoop.hbase.coprocessor. MultiRowMutationEndpoint] > {code} > This led to rst.join() hanging: > {code} > "RS:0;cn012:49371" #648 prio=5 os_prio=0 tid=0x7fdab24b5000 nid=0x621a > waiting on condition [0x7fd785fe] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1326) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1312) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1082) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16349) TestClusterId may hang during cluster shutdown
[ https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16349: --- Resolution: Fixed Assignee: Ted Yu Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 Status: Resolved (was: Patch Available) The test passed in recent builds. Will reopen if it fails in shutdown. Thanks for the review, Heng. > TestClusterId may hang during cluster shutdown > -- > > Key: HBASE-16349 > URL: https://issues.apache.org/jira/browse/HBASE-16349 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: 16349.branch-1.v1.txt, 16349.v1.txt > > > I was running TestClusterId on branch-1 where I observed the test hang during > test tearDown(). > {code} > 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(1415): Closing hbase:meta,,1.1588230740: disabling > compactions & flushes > 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(1442): Updates disabled for region > hbase:meta,,1.1588230740 > 2016-08-03 11:36:39,600 INFO [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegion(2253): Flushing 1/1 column families, memstore=232 B > 2016-08-03 11:36:39,601 WARN [RS_OPEN_META-cn012:49371-0.append-pool17-t1] > wal.FSHLog$RingBufferEventHandler(1900): Append sequenceId=8, requesting roll > of WAL > java.io.IOException: All datanodes > DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK] > are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402) > 2016-08-03 11:36:39,602 FATAL [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegionServer(2085): ABORTING region server > cn012.l42scl.hortonworks.com,49371,1470249187586: Unrecoverable > exception while closing region hbase:meta,,1.1588230740, still finishing close > org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append > sequenceId=8, requesting roll of WAL > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1902) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1754) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1676) > at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: All datanodes > DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK] > are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402) > 2016-08-03 11:36:39,603 FATAL [RS_CLOSE_META-cn012:49371-0] > regionserver.HRegionServer(2093): RegionServer abort: loaded coprocessors > are: [org.apache.hadoop.hbase.coprocessor. MultiRowMutationEndpoint] > {code} > This led to rst.join() hanging: > {code} > "RS:0;cn012:49371" #648 prio=5 os_prio=0 tid=0x7fdab24b5000 nid=0x621a > waiting on condition [0x7fd785fe] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1326) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1312) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1082) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16578) Mob data loss after mob compaction and normal compcation
[ https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497684#comment-15497684 ] huaxiang sun commented on HBASE-16578: -- Sorry [~jingcheng...@intel.com]. I do not have bandwidth to work on this issue recently. Could you go ahead to post the patch? Thanks. > Mob data loss after mob compaction and normal compcation > > > Key: HBASE-16578 > URL: https://issues.apache.org/jira/browse/HBASE-16578 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: TestMobCompaction.java, TestMobCompaction.java > > > I have a unittest case, which could explore the mob data loss issue. The root > cause is that with the following line: > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java#L625 > It will make the old mob reference cell win during compaction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497687#comment-15497687 ] Ted Yu commented on HBASE-16646: In the map: bq. Map> map key is family. value is List of Paths each of which contains family. > Enhance LoadIncrementalHFiles to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497689#comment-15497689 ] huaxiang sun commented on HBASE-16345: -- Hi [~enis], any comments for v5 patch? Thanks. > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16345-v001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, > HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, > HBASE-16345.master.005.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16647) hbck should do offline reference repair before online repair
[ https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16647: - Attachment: HBASE-16647.patch > hbck should do offline reference repair before online repair > > > Key: HBASE-16647 > URL: https://issues.apache.org/jira/browse/HBASE-16647 > Project: HBase > Issue Type: Bug >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-16647.patch > > > {noformat} > hbck > -fixReferenceFiles Try to offline lingering reference store files > Metadata Repair shortcuts > -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles > -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps > -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes > {noformat} > Bad reference files prevent the region from coming online. > If used in the shortcut combination, the reference files should be fixed > before other online fix. > I have seen repeated '-repair' did not work because bad reference files > failed regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16647) hbck should do offline reference repair before online repair
[ https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-16647: - Status: Patch Available (was: Open) > hbck should do offline reference repair before online repair > > > Key: HBASE-16647 > URL: https://issues.apache.org/jira/browse/HBASE-16647 > Project: HBase > Issue Type: Bug >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-16647.patch > > > {noformat} > hbck > -fixReferenceFiles Try to offline lingering reference store files > Metadata Repair shortcuts > -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles > -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps > -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes > {noformat} > Bad reference files prevent the region from coming online. > If used in the shortcut combination, the reference files should be fixed > before other online fix. > I have seen repeated '-repair' did not work because bad reference files > failed regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497701#comment-15497701 ] Jerry He commented on HBASE-16257: -- Good point.Let me think thru this case. FYI Ashish Singhi > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497701#comment-15497701 ] Jerry He edited comment on HBASE-16257 at 9/16/16 11:16 PM: Good point.Let me think thru this case. FYI [~ashish singhi] was (Author: jinghe): Good point.Let me think thru this case. FYI Ashish Singhi > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16534) Procedure v2 - Perf Tool for Scheduler
[ https://issues.apache.org/jira/browse/HBASE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16534: - Attachment: HBASE-16534.master.001.patch > Procedure v2 - Perf Tool for Scheduler > -- > > Key: HBASE-16534 > URL: https://issues.apache.org/jira/browse/HBASE-16534 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16534.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497740#comment-15497740 ] Vladimir Rodionov commented on HBASE-16620: --- [~tedyu] can you fix compilation error? BackupDriver? > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > 16620.addendum4, HBASE-16620-v1.patch, HBASE-16620-v3.patch, > HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Summary: Enhance LoadIncrementalHFiles API to accept store file paths as input (was: Enhance LoadIncrementalHFiles to accept store file paths as input) > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16620: --- Attachment: 16620.addendum5 Compilation has been fixed. > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > 16620.addendum4, 16620.addendum5, HBASE-16620-v1.patch, HBASE-16620-v3.patch, > HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16534) Procedure v2 - Perf Tool for Scheduler
[ https://issues.apache.org/jira/browse/HBASE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16534: - Attachment: HBASE-16534.master.002.patch > Procedure v2 - Perf Tool for Scheduler > -- > > Key: HBASE-16534 > URL: https://issues.apache.org/jira/browse/HBASE-16534 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16534.master.001.patch, > HBASE-16534.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[ https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15448: -- Attachment: HBASE-15448-v5.patch v5. Modified TestIncrementalBackup to perform region splits between full and incremental backup to test new full backup restore logic (when # of snapshot regions != # of regions in a table we restore data into) > HBase Backup Phase 3: Restore optimization 2 > > > Key: HBASE-15448 > URL: https://issues.apache.org/jira/browse/HBASE-15448 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch, > HBASE-15448-v4.patch, HBASE-15448-v5.patch > > > JIRA opened to continue work on restore optimization. > This will focus on the following > # During incremental backup image restore - restoring full backup into region > boundaries of the most recent incremental backup image. > # Combining multiple tables into single M/R job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[ https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15448: -- Attachment: (was: HBASE-15448-v5.patch) > HBase Backup Phase 3: Restore optimization 2 > > > Key: HBASE-15448 > URL: https://issues.apache.org/jira/browse/HBASE-15448 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch, > HBASE-15448-v4.patch, HBASE-15448-v5.patch > > > JIRA opened to continue work on restore optimization. > This will focus on the following > # During incremental backup image restore - restoring full backup into region > boundaries of the most recent incremental backup image. > # Combining multiple tables into single M/R job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[ https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15448: -- Attachment: HBASE-15448-v5.patch > HBase Backup Phase 3: Restore optimization 2 > > > Key: HBASE-15448 > URL: https://issues.apache.org/jira/browse/HBASE-15448 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch, > HBASE-15448-v4.patch, HBASE-15448-v5.patch > > > JIRA opened to continue work on restore optimization. > This will focus on the following > # During incremental backup image restore - restoring full backup into region > boundaries of the most recent incremental backup image. > # Combining multiple tables into single M/R job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[ https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497893#comment-15497893 ] Hadoop QA commented on HBASE-15448: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s {color} | {color:red} HBASE-15448 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828945/HBASE-15448-v5.patch | | JIRA Issue | HBASE-15448 | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/3579/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > HBase Backup Phase 3: Restore optimization 2 > > > Key: HBASE-15448 > URL: https://issues.apache.org/jira/browse/HBASE-15448 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch, > HBASE-15448-v4.patch, HBASE-15448-v5.patch > > > JIRA opened to continue work on restore optimization. > This will focus on the following > # During incremental backup image restore - restoring full backup into region > boundaries of the most recent incremental backup image. > # Combining multiple tables into single M/R job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Status: Patch Available (was: Open) > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16534) Procedure v2 - Perf Tool for Scheduler
[ https://issues.apache.org/jira/browse/HBASE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16534: Status: Patch Available (was: Open) > Procedure v2 - Perf Tool for Scheduler > -- > > Key: HBASE-16534 > URL: https://issues.apache.org/jira/browse/HBASE-16534 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16534.master.001.patch, > HBASE-16534.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16534) Procedure v2 - Perf Tool for Scheduler
[ https://issues.apache.org/jira/browse/HBASE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16534: Fix Version/s: (was: 1.4.0) > Procedure v2 - Perf Tool for Scheduler > -- > > Key: HBASE-16534 > URL: https://issues.apache.org/jira/browse/HBASE-16534 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16534.master.001.patch, > HBASE-16534.master.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair
[ https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497932#comment-15497932 ] Hadoop QA commented on HBASE-16647: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 19s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 48s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 27s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 53s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 135m 23s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.util.TestHBaseFsckTwoRS | | Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide | | | org.apache.hadoop.hbase.client.TestEnableTable | | | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas | | | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient | | | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient | | | org.apache.hadoop.hbase.client.TestIncrementsFromClientSide | | | org.apache.hadoop.hbase.client.TestResultSizeEstimation | | | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828927/HBASE-16647.patch | | JIRA Issue | HBASE-16647 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a336abed431e 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 1a1003a | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/3578
[jira] [Commented] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide
[ https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497934#comment-15497934 ] Jerry He commented on HBASE-16544: -- Committed the addendum. Thanks. > Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide > --- > > Key: HBASE-16544 > URL: https://issues.apache.org/jira/browse/HBASE-16544 > Project: HBase > Issue Type: Bug > Components: documentation, Filesystem Integration >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16544-V1.patch, HBASE-16544-addendum.patch > > > reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration > (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I > think this title means that we can use S3 storage to replace HDFS, I really > tried this :(, always give me errors and hbase even can not start, see > error mentioned in jira HBASE-11045. > (2) And the details in this section are more about deploying HBase on Amazon > EC2 cluster, which has nothing to do 'using Amazon S3 storage' > (3) In all, I think we need to remove this section, or at least clarify this > section if someone fully test HBase on S3. see HBASE-15646 for more details > about this doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498070#comment-15498070 ] Jerry He commented on HBASE-16257: -- Hi, [~enis] Thinking a bit more. Correct me if I am wrong. HFileReplicator operates on the sink side. It tries to copy from the sourceFs to local sink staging dir. If we have 2.x -> 1.x, there is no rootDir/staging on the sink. But it is ok. Sink knows how to handle its own staging dir, new way or old way. Source will not try to access sink staging dir and sink will not try to access source staging dir. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16534) Procedure v2 - Perf Tool for Scheduler
[ https://issues.apache.org/jira/browse/HBASE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498073#comment-15498073 ] Hadoop QA commented on HBASE-16534: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 51s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 22s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 59s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 24m 42s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s {color} | {color:green} hbase-procedure in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 48s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 117m 11s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure | | | org.apache.hadoop.hbase.replication.TestReplicationStateHBaseImpl | | | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures | | | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache | | | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828941/HBASE-16534.master.002.patch | | JIRA Issue | HBASE-16534 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 04d88e67435b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 1a1003a | | Default J
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498088#comment-15498088 ] Hadoop QA commented on HBASE-16646: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 5s {color} | {color:blue} The patch file was not named according to hbase's naming conventions. Please see https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for instructions. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 42s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 57s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 46s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s {color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 29s {color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 21s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 139m 18s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hbase-server | | | Nullcheck of queue at line 478 of value previously dereferenced in org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.performBulkLoad(Admin, Table, RegionLocator, Deque) At LoadIncrementalHFiles.java:478 of value previously dereferenced in org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.performBulkLoad(Admin, Table, RegionLocator, Deque) At LoadIncrementalHFiles.java:[line 427] | | Failed junit tests | hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures | | | hadoop.hbase.regionserver.TestHRegion | | Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient | | | org.apache.hadoop.hbase.wal.TestWALFiltering | | | org.apache.hadoop.hbase.wal.TestWALSplitCompressed | | | org.apache.hadoop.hbase.TestZooKeeper | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828925/16646.v1.txt | | JIRA Issue
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Attachment: 16646.v2.txt > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt, 16646.v2.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498121#comment-15498121 ] Ted Yu commented on HBASE-16646: Patch v2 fixes javadoc and findbugs warnings. > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt, 16646.v2.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair
[ https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498159#comment-15498159 ] Ted Yu commented on HBASE-16647: Looks like testLingeringReferenceFile needs to be adjusted for the change. > hbck should do offline reference repair before online repair > > > Key: HBASE-16647 > URL: https://issues.apache.org/jira/browse/HBASE-16647 > Project: HBase > Issue Type: Bug >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-16647.patch > > > {noformat} > hbck > -fixReferenceFiles Try to offline lingering reference store files > Metadata Repair shortcuts > -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles > -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps > -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes > {noformat} > Bad reference files prevent the region from coming online. > If used in the shortcut combination, the reference files should be fixed > before other online fix. > I have seen repeated '-repair' did not work because bad reference files > failed regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide
[ https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498188#comment-15498188 ] Hudson commented on HBASE-16544: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1619 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1619/]) HBASE-16544 Remove or Clarify Using Amazon S3 Storage section in the (jerryjch: rev bb3d9ccd489fb64e3cb2020583935a393382a678) * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc > Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide > --- > > Key: HBASE-16544 > URL: https://issues.apache.org/jira/browse/HBASE-16544 > Project: HBase > Issue Type: Bug > Components: documentation, Filesystem Integration >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-16544-V1.patch, HBASE-16544-addendum.patch > > > reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration > (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I > think this title means that we can use S3 storage to replace HDFS, I really > tried this :(, always give me errors and hbase even can not start, see > error mentioned in jira HBASE-11045. > (2) And the details in this section are more about deploying HBase on Amazon > EC2 cluster, which has nothing to do 'using Amazon S3 storage' > (3) In all, I think we need to remove this section, or at least clarify this > section if someone fully test HBase on S3. see HBASE-15646 for more details > about this doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498196#comment-15498196 ] Ashish Singhi commented on HBASE-16598: --- +1 > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, > HBASE-16598-v4.patch, HBASE-16598.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair
[ https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498211#comment-15498211 ] Jerry He commented on HBASE-16647: -- The error state gets cleared in later stages after the move of offlineReferenceFileRepair to the beginning. Will fix the test. Thanks, Ted. > hbck should do offline reference repair before online repair > > > Key: HBASE-16647 > URL: https://issues.apache.org/jira/browse/HBASE-16647 > Project: HBase > Issue Type: Bug >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-16647.patch > > > {noformat} > hbck > -fixReferenceFiles Try to offline lingering reference store files > Metadata Repair shortcuts > -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles > -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps > -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes > {noformat} > Bad reference files prevent the region from coming online. > If used in the shortcut combination, the reference files should be fixed > before other online fix. > I have seen repeated '-repair' did not work because bad reference files > failed regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)