[jira] [Commented] (HBASE-20984) Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
[
https://issues.apache.org/jira/browse/HBASE-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722023#comment-16722023
]
Sakthi commented on HBASE-20984:
Have uploaded another patch with fixes for the compile & checkstyle issues.
> Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
> --
>
> Key: HBASE-20984
> URL: https://issues.apache.org/jira/browse/HBASE-20984
> Project: HBase
> Issue Type: Bug
> Components: test, wal
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20984.master.001.patch,
> hbase-20984.master.002.patch
>
>
> The current setup in TestWALFactory tries to create custom WAL directory
> outside hdfs but ends up creating a custom WAL directory inside hdfs. In
> TestWALFactory.java:
> {code:java}
> public static void setUpBeforeClass() throws Exception {
> CommonFSUtils.setWALRootDir(TEST_UTIL.getConfiguration(), new
> Path("file:///tmp/wal")); // A local filesystem WAL is attempted
> ...
> hbaseDir = TEST_UTIL.createRootDir();
> hbaseWALDir = TEST_UTIL.createWALRootDir(); // But a directory inside
> hdfs is created here using HBaseTestingUtility#getNewDataTestDirOnTestFS
> }
> {code}
> The change was made in HBASE-20723
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20984) Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
[
https://issues.apache.org/jira/browse/HBASE-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-20984:
---
Attachment: hbase-20984.master.002.patch
> Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
> --
>
> Key: HBASE-20984
> URL: https://issues.apache.org/jira/browse/HBASE-20984
> Project: HBase
> Issue Type: Bug
> Components: test, wal
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20984.master.001.patch,
> hbase-20984.master.002.patch
>
>
> The current setup in TestWALFactory tries to create custom WAL directory
> outside hdfs but ends up creating a custom WAL directory inside hdfs. In
> TestWALFactory.java:
> {code:java}
> public static void setUpBeforeClass() throws Exception {
> CommonFSUtils.setWALRootDir(TEST_UTIL.getConfiguration(), new
> Path("file:///tmp/wal")); // A local filesystem WAL is attempted
> ...
> hbaseDir = TEST_UTIL.createRootDir();
> hbaseWALDir = TEST_UTIL.createWALRootDir(); // But a directory inside
> hdfs is created here using HBaseTestingUtility#getNewDataTestDirOnTestFS
> }
> {code}
> The change was made in HBASE-20723
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20984) Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
[
https://issues.apache.org/jira/browse/HBASE-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722019#comment-16722019
]
Hadoop QA commented on HBASE-20984:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
23s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 23s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
11s{color} | {color:red} hbase-server: The patch generated 1 new + 244
unchanged - 0 fixed = 245 total (was 244) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 23s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}123m
58s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20984 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951874/hbase-20984.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0f712dc73adc 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 491153488e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| compile |
https://builds.apache.org/job/PreCommit-HBASE-Build/15289/artifact/patchprocess/patch-compile-hbase-server.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/15289/artifact/patchprocess/patch-compile-hbase-server.txt
|
| checkstyle |
[jira] [Commented] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[
https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721996#comment-16721996
]
Hudson commented on HBASE-21590:
Results for branch branch-2.0
[build #1165 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1165/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1165//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1165//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1165//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Optimize trySkipToNextColumn in StoreScanner a bit
> --
>
> Key: HBASE-21590
> URL: https://issues.apache.org/jira/browse/HBASE-21590
> Project: HBase
> Issue Type: Improvement
> Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10
>
> Attachments: 21590-1.5.txt, HBASE-21590-master.txt,
> HBASE-21590.addendum.v1.patch
>
>
> See latest comment on HBASE-17958
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20917) MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
[
https://issues.apache.org/jira/browse/HBASE-20917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721995#comment-16721995
]
Hudson commented on HBASE-20917:
Results for branch branch-2.0
[build #1165 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1165/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1165//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1165//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1165//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
> --
>
> Key: HBASE-20917
> URL: https://issues.apache.org/jira/browse/HBASE-20917
> Project: HBase
> Issue Type: Bug
> Components: meta, metrics
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0
>
> Attachments: 20917.addendum, 20917.v1.txt, 20917.v2.txt
>
>
> I noticed the following in test output:
> {code}
> 2018-07-21 15:54:43,181 ERROR [RS_CLOSE_REGION-regionserver/172.17.5.4:0-1]
> executor.EventHandler(186): Caught throwable while processing event
> M_RS_CLOSE_REGION
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics.stop(MetaTableMetrics.java:329)
> at
> org.apache.hadoop.hbase.coprocessor.BaseEnvironment.shutdown(BaseEnvironment.java:91)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment.shutdown(RegionCoprocessorHost.java:165)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:290)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.postEnvCall(RegionCoprocessorHost.java:559)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:622)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postClose(RegionCoprocessorHost.java:551)
> at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1678)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1484)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> {code}
> {{requestsMap}} is only initialized for the meta region.
> However, check for meta region is absent in the stop method:
> {code}
> public void stop(CoprocessorEnvironment e) throws IOException {
> // since meta region can move around, clear stale metrics when stop.
> for (String meterName : requestsMap.keySet()) {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721993#comment-16721993 ] Lars Hofhansl commented on HBASE-21590: --- Sorry about the checkstyle stuff, and thanks for fixing [~busbey]. > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt, > HBASE-21590.addendum.v1.patch > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20984) Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
[
https://issues.apache.org/jira/browse/HBASE-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721966#comment-16721966
]
Sakthi commented on HBASE-20984:
Hi [~busbey], I have removed the unneeded line in TestWALFactory and have
modified TestWALRootDir to test custom WALs in multiple filesystems(hdfs &
local). Also have utilized the existing temp folder creation utilities to get
one for local as well.
I did verify that local WALs were being created by the test.
> Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
> --
>
> Key: HBASE-20984
> URL: https://issues.apache.org/jira/browse/HBASE-20984
> Project: HBase
> Issue Type: Bug
> Components: test, wal
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20984.master.001.patch
>
>
> The current setup in TestWALFactory tries to create custom WAL directory
> outside hdfs but ends up creating a custom WAL directory inside hdfs. In
> TestWALFactory.java:
> {code:java}
> public static void setUpBeforeClass() throws Exception {
> CommonFSUtils.setWALRootDir(TEST_UTIL.getConfiguration(), new
> Path("file:///tmp/wal")); // A local filesystem WAL is attempted
> ...
> hbaseDir = TEST_UTIL.createRootDir();
> hbaseWALDir = TEST_UTIL.createWALRootDir(); // But a directory inside
> hdfs is created here using HBaseTestingUtility#getNewDataTestDirOnTestFS
> }
> {code}
> The change was made in HBASE-20723
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20984) Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
[
https://issues.apache.org/jira/browse/HBASE-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-20984:
---
Attachment: hbase-20984.master.001.patch
> Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
> --
>
> Key: HBASE-20984
> URL: https://issues.apache.org/jira/browse/HBASE-20984
> Project: HBase
> Issue Type: Bug
> Components: test, wal
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20984.master.001.patch
>
>
> The current setup in TestWALFactory tries to create custom WAL directory
> outside hdfs but ends up creating a custom WAL directory inside hdfs. In
> TestWALFactory.java:
> {code:java}
> public static void setUpBeforeClass() throws Exception {
> CommonFSUtils.setWALRootDir(TEST_UTIL.getConfiguration(), new
> Path("file:///tmp/wal")); // A local filesystem WAL is attempted
> ...
> hbaseDir = TEST_UTIL.createRootDir();
> hbaseWALDir = TEST_UTIL.createWALRootDir(); // But a directory inside
> hdfs is created here using HBaseTestingUtility#getNewDataTestDirOnTestFS
> }
> {code}
> The change was made in HBASE-20723
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HBASE-20984) Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
[
https://issues.apache.org/jira/browse/HBASE-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-20984 started by Sakthi.
--
> Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
> --
>
> Key: HBASE-20984
> URL: https://issues.apache.org/jira/browse/HBASE-20984
> Project: HBase
> Issue Type: Bug
> Components: test, wal
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20984.master.001.patch
>
>
> The current setup in TestWALFactory tries to create custom WAL directory
> outside hdfs but ends up creating a custom WAL directory inside hdfs. In
> TestWALFactory.java:
> {code:java}
> public static void setUpBeforeClass() throws Exception {
> CommonFSUtils.setWALRootDir(TEST_UTIL.getConfiguration(), new
> Path("file:///tmp/wal")); // A local filesystem WAL is attempted
> ...
> hbaseDir = TEST_UTIL.createRootDir();
> hbaseWALDir = TEST_UTIL.createWALRootDir(); // But a directory inside
> hdfs is created here using HBaseTestingUtility#getNewDataTestDirOnTestFS
> }
> {code}
> The change was made in HBASE-20723
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20984) Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
[
https://issues.apache.org/jira/browse/HBASE-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-20984:
---
Status: Patch Available (was: In Progress)
> Add/Modify test case to check custom hbase.wal.dir outside hdfs filesystem
> --
>
> Key: HBASE-20984
> URL: https://issues.apache.org/jira/browse/HBASE-20984
> Project: HBase
> Issue Type: Bug
> Components: test, wal
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20984.master.001.patch
>
>
> The current setup in TestWALFactory tries to create custom WAL directory
> outside hdfs but ends up creating a custom WAL directory inside hdfs. In
> TestWALFactory.java:
> {code:java}
> public static void setUpBeforeClass() throws Exception {
> CommonFSUtils.setWALRootDir(TEST_UTIL.getConfiguration(), new
> Path("file:///tmp/wal")); // A local filesystem WAL is attempted
> ...
> hbaseDir = TEST_UTIL.createRootDir();
> hbaseWALDir = TEST_UTIL.createWALRootDir(); // But a directory inside
> hdfs is created here using HBaseTestingUtility#getNewDataTestDirOnTestFS
> }
> {code}
> The change was made in HBASE-20723
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21607) HBase Region server read fails in case of datanode disk error.
[
https://issues.apache.org/jira/browse/HBASE-21607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721916#comment-16721916
]
Lars Hofhansl commented on HBASE-21607:
---
I guess the question I have: Why is there a NULL-checksum option in the first
place? We turned on HBase checksums, and there's an option to configure a no-op
checksum... Seems at least somewhat pointless. :)
> HBase Region server read fails in case of datanode disk error.
> --
>
> Key: HBASE-21607
> URL: https://issues.apache.org/jira/browse/HBASE-21607
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.3.2
>Reporter: Rushabh S Shah
>Priority: Major
>
> Hbase region server reads failed with following error.
> {noformat}
> 2018-11-30 16:49:18,760 WARN [,queue=12,port=60020] hdfs.BlockReaderFactory -
> BlockReaderFactory(fileName=,
> block=BP-1618467445--1516873463430:blk_1090719164_16982933): error
> creating ShortCircuitReplica.
> java.io.IOException: invalid metadata header version 0. Can only handle
> version 1.
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica.(ShortCircuitReplica.java:129)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:558)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:490)
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:782)
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:716)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
> at
> org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1145)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1087)
> at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1444)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1407)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:89)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:834)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1530)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1781)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1624)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:455)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1263)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:297)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:189)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:372)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:220)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2164)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5916)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5890)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2739)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2719)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7197)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7156)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7149)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2250)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:35068)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2373)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>
> 2018-11-30 16:49:18,760 WARN [,queue=12,port=60020]
> shortcircuit.ShortCircuitCache - ShortCircuitCache(0x16fd768f): failed to
> load 1090719164_BP-1618467445--1516873463430
> 2018-11-30 16:49:18,761 DEBUG [,queue=12,port=60020] ipc.RpcServer -
> RpcServer.FifoWFPBQ.default.handler=246,queue=12,port=60020: callId: 46940
> service: ClientService methodName: Get size: 443 connection: :48798
> deadline: 1543596678759
> 2018-11-30 16:49:18,761 DEBUG
[jira] [Updated] (HBASE-21592) quota.addGetResult(r) throw NPE
[
https://issues.apache.org/jira/browse/HBASE-21592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xuqinya updated HBASE-21592:
Affects Version/s: (was: 1.2.9)
(was: 2.0.2)
> quota.addGetResult(r) throw NPE
> -
>
> Key: HBASE-21592
> URL: https://issues.apache.org/jira/browse/HBASE-21592
> Project: HBase
> Issue Type: Bug
>Reporter: xuqinya
>Assignee: xuqinya
>Priority: Major
> Attachments: HBASE-21592.master.0001.patch
>
>
> Setting the RPC quota, table.exists(Get) cause quota.addGetResult(r) throw
> NPE.
>
> set_quota TYPE => THROTTLE, NAMESPACE => 'ns1', LIMIT => '1000req/sec'
> Connection conn = ConnectionFactory.createConnection(config);
> Table htable = conn.getTable(TableName.valueOf("ns1:t1"));
> boolean exists = htable.exists(new Get(Bytes.toBytes("123")));
> log:
> java.io.IOException: java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2183)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.quotas.QuotaUtil.calculateResultSize(QuotaUtil.java:282)
> at
> org.apache.hadoop.hbase.quotas.DefaultOperationQuota.addGetResult(DefaultOperationQuota.java:99)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1907)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2135)
> ... 4 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21607) HBase Region server read fails in case of datanode disk error.
[
https://issues.apache.org/jira/browse/HBASE-21607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721914#comment-16721914
]
Lars Hofhansl commented on HBASE-21607:
---
This of course is rare case... Choosing 0 read from a datastore is just bad
practice, though. Not sure how we can change it.
I suppose we could have a new HFile version and have 0 indicate an error and
pick only non-zero (and non-255) value.
> HBase Region server read fails in case of datanode disk error.
> --
>
> Key: HBASE-21607
> URL: https://issues.apache.org/jira/browse/HBASE-21607
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.3.2
>Reporter: Rushabh S Shah
>Priority: Major
>
> Hbase region server reads failed with following error.
> {noformat}
> 2018-11-30 16:49:18,760 WARN [,queue=12,port=60020] hdfs.BlockReaderFactory -
> BlockReaderFactory(fileName=,
> block=BP-1618467445--1516873463430:blk_1090719164_16982933): error
> creating ShortCircuitReplica.
> java.io.IOException: invalid metadata header version 0. Can only handle
> version 1.
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica.(ShortCircuitReplica.java:129)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:558)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:490)
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:782)
> at
> org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:716)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
> at
> org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1145)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1087)
> at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1444)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1407)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:89)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:834)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1530)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1781)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1624)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:455)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1263)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:297)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:189)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:372)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:220)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2164)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5916)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5890)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2739)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2719)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7197)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7156)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7149)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2250)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:35068)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2373)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>
> 2018-11-30 16:49:18,760 WARN [,queue=12,port=60020]
> shortcircuit.ShortCircuitCache - ShortCircuitCache(0x16fd768f): failed to
> load 1090719164_BP-1618467445--1516873463430
> 2018-11-30 16:49:18,761 DEBUG [,queue=12,port=60020] ipc.RpcServer -
> RpcServer.FifoWFPBQ.default.handler=246,queue=12,port=60020: callId: 46940
> service: ClientService methodName: Get size: 443 connection: :48798
> deadline: 1543596678759
> 2018-11-30
[jira] [Commented] (HBASE-21565) Delete dead server from dead server list too early leads to concurrent Server Crash Procedures(SCP) for a same server
[
https://issues.apache.org/jira/browse/HBASE-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721913#comment-16721913
]
Jingyun Tian commented on HBASE-21565:
--
[~stack]
{quote}Its a big change setting holdLock on SCP?
{quote}
Not really a big change I think, since only one kind procedure
ServerCrashProcedure will grab a lock of a dead server, it only used to prevent
two SCPs for the same server running concurrently.
{quote}// '-SPLITTING'. Each splitting server should have a corresponding SCP.
Log if not.
{quote}
I'll upload a patch later to fix this.
> Delete dead server from dead server list too early leads to concurrent Server
> Crash Procedures(SCP) for a same server
> -
>
> Key: HBASE-21565
> URL: https://issues.apache.org/jira/browse/HBASE-21565
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Critical
> Attachments: HBASE-21565.master.001.patch,
> HBASE-21565.master.002.patch, HBASE-21565.master.003.patch,
> HBASE-21565.master.004.patch, HBASE-21565.master.005.patch,
> HBASE-21565.master.006.patch
>
>
> There are 2 kinds of SCP for a same server will be scheduled during cluster
> restart, one is ZK session timeout, the other one is new server report in
> will cause the stale one do fail over. The only barrier for these 2 kinds of
> SCP is check if the server is in the dead server list.
> {code}
> if (this.deadservers.isDeadServer(serverName)) {
> LOG.warn("Expiration called on {} but crash processing already in
> progress", serverName);
> return false;
> }
> {code}
> But the problem is when master finish initialization, it will delete all
> stale servers from dead server list. Thus when the SCP for ZK session timeout
> come in, the barrier is already removed.
> Here is the logs that how this problem occur.
> {code}
> 2018-12-07,11:42:37,589 INFO
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: Start pid=9,
> state=RUNNABLE:SERVER_CRASH_START, hasLock=true; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> 2018-12-07,11:42:58,007 INFO
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: Start pid=444,
> state=RUNNABLE:SERVER_CRASH_START, hasLock=true; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> {code}
> Now we can see two SCP are scheduled for the same server.
> But the first procedure is finished after the second SCP starts.
> {code}
> 2018-12-07,11:43:08,038 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=9,
> state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> in 30.5340sec
> {code}
> Thus it will leads the problem that regions will be assigned twice.
> {code}
> 2018-12-07,12:16:33,039 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: rit=OPEN,
> location=c4-hadoop-tst-st28.bj,29100,1544154149607, table=test_failover,
> region=459b3130b40caf3b8f3e1421766f4089 reported OPEN on
> server=c4-hadoop-tst-st29.bj,29100,1544154149615 but state has otherwise
> {code}
> And here we can see the server is removed from dead server list before the
> second SCP starts.
> {code}
> 2018-12-07,11:42:44,938 DEBUG org.apache.hadoop.hbase.master.DeadServer:
> Removed c4-hadoop-tst-st27.bj,29100,1544153846859 ; numProcessing=3
> {code}
> Thus we should not delete dead server from dead server list immediately.
> Patch to fix this problem will be upload later.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21504) If enable FIFOCompactionPolicy, a compaction may write a "empty" hfile whose maxTimeStamp is long max. This kind of hfile will never be archived.
[ https://issues.apache.org/jira/browse/HBASE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721892#comment-16721892 ] Hudson commented on HBASE-21504: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #512 (See [https://builds.apache.org/job/HBase-1.3-IT/512/]) Revert "HBASE-21504 If enable FIFOCompactionPolicy, a compaction may (apurtell: rev d2c1deef1c1dd858fcf0b110753e57be466e676f) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/FIFOCompactionPolicy.java > If enable FIFOCompactionPolicy, a compaction may write a "empty" hfile whose > maxTimeStamp is long max. This kind of hfile will never be archived. > - > > Key: HBASE-21504 > URL: https://issues.apache.org/jira/browse/HBASE-21504 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.1.0 >Reporter: xuming >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.9, 2.1.2, 2.0.4 > > Attachments: 1.patch, HBASE-21504.v1.patch > > > When i use FIFOCompactionPolicy, and if all hfiles(>1) are TTL expired in a > region, once we do a compaction on the region, the compaction policy will > select the latest hfile to do compaction.But beacuse the latest hfile is > already TTL expired, compactor only write a "empty" hfile(whose entry counter > is 0 and maxTimeStamp is Long.MAX_VALUE) finally. Because maxTimeStamp is > long max, so the "empty" hfile will never be TTL expired, more seriously we > can not archive it by FIFOCompactionPolicy forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21504) If enable FIFOCompactionPolicy, a compaction may write a "empty" hfile whose maxTimeStamp is long max. This kind of hfile will never be archived.
[ https://issues.apache.org/jira/browse/HBASE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21504: --- Fix Version/s: (was: 1.3.3) > If enable FIFOCompactionPolicy, a compaction may write a "empty" hfile whose > maxTimeStamp is long max. This kind of hfile will never be archived. > - > > Key: HBASE-21504 > URL: https://issues.apache.org/jira/browse/HBASE-21504 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 2.1.0 >Reporter: xuming >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.9, 2.1.2, 2.0.4 > > Attachments: 1.patch, HBASE-21504.v1.patch > > > When i use FIFOCompactionPolicy, and if all hfiles(>1) are TTL expired in a > region, once we do a compaction on the region, the compaction policy will > select the latest hfile to do compaction.But beacuse the latest hfile is > already TTL expired, compactor only write a "empty" hfile(whose entry counter > is 0 and maxTimeStamp is Long.MAX_VALUE) finally. Because maxTimeStamp is > long max, so the "empty" hfile will never be TTL expired, more seriously we > can not archive it by FIFOCompactionPolicy forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721874#comment-16721874
]
Hudson commented on HBASE-21578:
Results for branch branch-2
[build #1558 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1558/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1558//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1558//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1558//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix wrong throttling exception for capacity unit
>
>
> Key: HBASE-21578
> URL: https://issues.apache.org/jira/browse/HBASE-21578
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21578.master.001.patch,
> HBASE-21578.master.002.patch
>
>
> HBASE-21034 provides a new throttle type: capacity unit, but the throttling
> exception is confusing:
>
> {noformat}
> 2018-12-11 14:38:41,503 DEBUG [Time-limited test]
> client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
> started=0 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write size limit
> exceeded - wait 10sec
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:106)
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteSizeExceeded(RpcThrottlingException.java:96)
> at
> org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:179){noformat}
> Need to make the exception more clearly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21590: Resolution: Fixed Status: Resolved (was: Patch Available) patch pushed to branch-2.1 and branc-2.0. addendum pushed to all branches. > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt, > HBASE-21590.addendum.v1.patch > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21358) Snapshot procedure fails but SnapshotManager thinks it is still running
[
https://issues.apache.org/jira/browse/HBASE-21358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721824#comment-16721824
]
Andrew Purtell commented on HBASE-21358:
[~taklwu] Sorry for the delayed response. What I have reported is your case #2.
To reproduce start a snapshot and cause chaos on the cluster by killing a
process or moving a region. The slowDeterminstic chaos policy can do this for
you. The condition does eventually resolve. It is working as designed but I do
think it could fail faster.
Your case #1 is interesting. If you still want to take this, please go ahead,
fixing a race is always good (case #1) and if we can get case #2 to fail faster
that would be good too.
> Snapshot procedure fails but SnapshotManager thinks it is still running
> ---
>
> Key: HBASE-21358
> URL: https://issues.apache.org/jira/browse/HBASE-21358
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 1.3.2
>Reporter: Andrew Purtell
>Priority: Major
>
> A snapshot procedure fails due to chaotic test action but the snapshot
> manager still thinks it is running. The test client spins needlessly checking
> for something that will never actually complete. We give up eventually but we
> could be failing this a lot faster.
> On the integration client we are checking and re-checking:
> 2018-10-20 01:06:11,718 DEBUG [ChaosMonkeyThread] client.HBaseAdmin: Getting
> current status of snapshot from master...
> 2018-10-20 01:06:11,719 DEBUG [ChaosMonkeyThread] client.HBaseAdmin: (#40)
> Sleeping: 8571ms while waiting for snapshot completion.
> This is what it looks like on the master side each time the client checks in:
> 2018-10-20 01:04:54,565 DEBUG
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=8100]
> master.MasterRpcServices: Checking to see if snapshot from request:{
> ss=IntegrationTestBigLinkedList-it-1539997289258
> table=IntegrationTestBigLinkedList type=FLUSH } is done
> 2018-10-20 01:04:54,565 DEBUG
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=8100]
> snapshot.SnapshotManager: Snapshoting '{
> ss=IntegrationTestBigLinkedList-it-1539997289258
> table=IntegrationTestBigLinkedList type=FLUSH }' is still in progress!
> There is no running procedure for the snapshot. The procedure has failed. The
> snapshot manager does not take any useful action afterward but believes the
> snapshot to still be in progress.
> I see related complaint from the hfile archiver task afterward, empty
> directories, failure to parse protobuf in descriptor files... Seems like
> there was junk in the filesystem left over from the failed snapshot. The
> master was soon restarted by chaos action, and now I don't see these
> complaints, so that partially complete snapshot may have been cleaned up.
> This is with 1.3.2, but patched to include the multithreaded hfile archiving
> improvements from later versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721817#comment-16721817
]
Hadoop QA commented on HBASE-19722:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
27s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
23s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 48s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}189m
56s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}234m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-19722 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951842/HBASE-19722-branch-2.1.v1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 25203b5c19af 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.1 / 0bb6f9e870 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
|
[jira] [Comment Edited] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721809#comment-16721809
]
Sean Busbey edited comment on HBASE-19722 at 12/14/18 9:43 PM:
---
[~stack] you want me to wait for HBASE-21606 before pushing this into banch-2.1
since you're in the middle of RCs? Or include now since it's already in 2.0 and
1.4 releases without the docs?
was (Author: busbey):
@stack you want me to wait for HBASE-21606 before pushing this into banch-2.1
since you're in the middle of RCs? Or include now since it's already in 2.0 and
1.4 releases without the docs?
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0, 2.0.2
>
> Attachments: HBASE-19722-branch-2.1.v1.patch,
> HBASE-19722.branch-1.v001.patch, HBASE-19722.branch-1.v002.patch,
> HBASE-19722.master.010.patch, HBASE-19722.master.011.patch,
> HBASE-19722.master.012.patch, HBASE-19722.master.013.patch,
> HBASE-19722.master.014.patch, HBASE-19722.master.015.patch,
> HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721809#comment-16721809
]
Sean Busbey commented on HBASE-19722:
-
@stack you want me to wait for HBASE-21606 before pushing this into banch-2.1
since you're in the middle of RCs? Or include now since it's already in 2.0 and
1.4 releases without the docs?
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0, 2.0.2
>
> Attachments: HBASE-19722-branch-2.1.v1.patch,
> HBASE-19722.branch-1.v001.patch, HBASE-19722.branch-1.v002.patch,
> HBASE-19722.master.010.patch, HBASE-19722.master.011.patch,
> HBASE-19722.master.012.patch, HBASE-19722.master.013.patch,
> HBASE-19722.master.014.patch, HBASE-19722.master.015.patch,
> HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21564) race condition in WAL rolling resulting in size-based rolling getting stuck
[
https://issues.apache.org/jira/browse/HBASE-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721792#comment-16721792
]
Sergey Shelukhin commented on HBASE-21564:
--
I wonder if it's some replication issue uncovered by this patch? WAL operations
seem to be the same between the successful and failed runs, however for some
reason replication for one file at the same offset produces different number of
edits (in the same test run):
{noformat}
Normal source for cluster 2: Total replicated edits: 1200, current progress:
walGroup [localhost%2C45311%2C1544660121842]: currently replicating from:
hdfs://localhost:42830/user/root/test-data/3c860f22-8402-6603-26e3-a2846cba30ef/WALs/localhost,45311,1544660121842/localhost%2C45311%2C1544660121842.1544660165345
at position: 88061
Normal source for cluster testInterClusterReplication: Total replicated edits:
946, current progress:
walGroup [localhost%2C45311%2C1544660121842]: currently replicating from:
hdfs://localhost:42830/user/root/test-data/3c860f22-8402-6603-26e3-a2846cba30ef/WALs/localhost,45311,1544660121842/localhost%2C45311%2C1544660121842.1544660165345
at position: 88061
{noformat}
> race condition in WAL rolling resulting in size-based rolling getting stuck
> ---
>
> Key: HBASE-21564
> URL: https://issues.apache.org/jira/browse/HBASE-21564
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HBASE-21564.master.001.patch,
> HBASE-21564.master.002.patch, HBASE-21564.master.003.patch,
> HBASE-21564.master.004.patch
>
>
> Manifests at least with AsyncFsWriter.
> There's a window after LogRoller replaces the writer in the WAL, but before
> it sets the rollLog boolean to false in the finally, where the WAL class can
> request another log roll (it can happen in particular when the logs are
> getting archived in the LogRoller thread, and there's high write volume
> causing the logs to roll quickly).
> LogRoller will blindly reset the rollLog flag in finally and "forget" about
> this request.
> AsyncWAL in turn never requests it again because its own rollRequested field
> is set and it expects a callback. Logs don't get rolled until a periodic roll
> is triggered after that.
> The acknowledgment of roll requests by LogRoller should be atomic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21535) Zombie Master detector is not working
[
https://issues.apache.org/jira/browse/HBASE-21535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721776#comment-16721776
]
stack commented on HBASE-21535:
---
All looks good. Pankaj, can you add an email to JIRA or else put up a version
of the patch that includes your email You could use the
./dev-support/submit-patch.py tool or just git format-patch... Thank you sir.
> Zombie Master detector is not working
> -
>
> Key: HBASE-21535
> URL: https://issues.apache.org/jira/browse/HBASE-21535
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21535.branch-2.patch, HBASE-21535.branch-2.patch,
> HBASE-21535.patch
>
>
> We have InitializationMonitor thread in HMaster which detects Zombie Hmaster
> based on _hbase.master.initializationmonitor.timeout _and halts if
> _hbase.master.initializationmonitor.haltontimeout_ set _true_.
> After HBASE-19694, HMaster initialization order was correted. Hmaster is set
> active after Initializing ZK system trackers as follows,
> {noformat}
> status.setStatus("Initializing ZK system trackers");
> initializeZKBasedSystemTrackers();
> status.setStatus("Loading last flushed sequence id of regions");
> try {
> this.serverManager.loadLastFlushedSequenceIds();
> } catch (IOException e) {
> LOG.debug("Failed to load last flushed sequence id of regions"
> + " from file system", e);
> }
> // Set ourselves as active Master now our claim has succeeded up in zk.
> this.activeMaster = true;
> {noformat}
> But Zombie detector thread is started at the begining phase of
> finishActiveMasterInitialization(),
> {noformat}
> private void finishActiveMasterInitialization(MonitoredTask status) throws
> IOException,
> InterruptedException, KeeperException, ReplicationException {
> Thread zombieDetector = new Thread(new InitializationMonitor(this),
> "ActiveMasterInitializationMonitor-" + System.currentTimeMillis());
> zombieDetector.setDaemon(true);
> zombieDetector.start();
> {noformat}
> During zombieDetector execution "master.isActiveMaster()" will be false, so
> it won't wait and cant detect zombie master.
> {noformat}
> @Override
> public void run() {
> try {
> while (!master.isStopped() && master.isActiveMaster()) {
> Thread.sleep(timeout);
> if (master.isInitialized()) {
> LOG.debug("Initialization completed within allotted tolerance. Monitor
> exiting.");
> } else {
> LOG.error("Master failed to complete initialization after " + timeout + "ms.
> Please"
> + " consider submitting a bug report including a thread dump of this
> process.");
> if (haltOnTimeout) {
> LOG.error("Zombie Master exiting. Thread dump to stdout");
> Threads.printThreadInfo(System.out, "Zombie HMaster");
> System.exit(-1);
> }
> }
> }
> } catch (InterruptedException ie) {
> LOG.trace("InitMonitor thread interrupted. Existing.");
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21565) Delete dead server from dead server list too early leads to concurrent Server Crash Procedures(SCP) for a same server
[
https://issues.apache.org/jira/browse/HBASE-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721769#comment-16721769
]
stack commented on HBASE-21565:
---
Nice find [~tianjingyun].
This log which is above your change needs updating... it seems like we now do
more than just log
// '-SPLITTING'. Each splitting server should have a corresponding SCP. Log if
not.
Its a big change setting holdLock on SCP?
Nice tests.
> Delete dead server from dead server list too early leads to concurrent Server
> Crash Procedures(SCP) for a same server
> -
>
> Key: HBASE-21565
> URL: https://issues.apache.org/jira/browse/HBASE-21565
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Critical
> Attachments: HBASE-21565.master.001.patch,
> HBASE-21565.master.002.patch, HBASE-21565.master.003.patch,
> HBASE-21565.master.004.patch, HBASE-21565.master.005.patch,
> HBASE-21565.master.006.patch
>
>
> There are 2 kinds of SCP for a same server will be scheduled during cluster
> restart, one is ZK session timeout, the other one is new server report in
> will cause the stale one do fail over. The only barrier for these 2 kinds of
> SCP is check if the server is in the dead server list.
> {code}
> if (this.deadservers.isDeadServer(serverName)) {
> LOG.warn("Expiration called on {} but crash processing already in
> progress", serverName);
> return false;
> }
> {code}
> But the problem is when master finish initialization, it will delete all
> stale servers from dead server list. Thus when the SCP for ZK session timeout
> come in, the barrier is already removed.
> Here is the logs that how this problem occur.
> {code}
> 2018-12-07,11:42:37,589 INFO
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: Start pid=9,
> state=RUNNABLE:SERVER_CRASH_START, hasLock=true; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> 2018-12-07,11:42:58,007 INFO
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: Start pid=444,
> state=RUNNABLE:SERVER_CRASH_START, hasLock=true; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> {code}
> Now we can see two SCP are scheduled for the same server.
> But the first procedure is finished after the second SCP starts.
> {code}
> 2018-12-07,11:43:08,038 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=9,
> state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> in 30.5340sec
> {code}
> Thus it will leads the problem that regions will be assigned twice.
> {code}
> 2018-12-07,12:16:33,039 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: rit=OPEN,
> location=c4-hadoop-tst-st28.bj,29100,1544154149607, table=test_failover,
> region=459b3130b40caf3b8f3e1421766f4089 reported OPEN on
> server=c4-hadoop-tst-st29.bj,29100,1544154149615 but state has otherwise
> {code}
> And here we can see the server is removed from dead server list before the
> second SCP starts.
> {code}
> 2018-12-07,11:42:44,938 DEBUG org.apache.hadoop.hbase.master.DeadServer:
> Removed c4-hadoop-tst-st27.bj,29100,1544153846859 ; numProcessing=3
> {code}
> Thus we should not delete dead server from dead server list immediately.
> Patch to fix this problem will be upload later.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[
https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721766#comment-16721766
]
Hadoop QA commented on HBASE-21590:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} hbase-server: The patch generated 0 new + 21
unchanged - 2 fixed = 21 total (was 23) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
19s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 37s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}127m
47s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951839/HBASE-21590.addendum.v1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 477d52479d49 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1b08ba7385 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15287/testReport/ |
| Max. process+thread count | 4171 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
|
[jira] [Created] (HBASE-21607) HBase Region server read fails in case of datanode disk error.
Rushabh S Shah created HBASE-21607:
--
Summary: HBase Region server read fails in case of datanode disk
error.
Key: HBASE-21607
URL: https://issues.apache.org/jira/browse/HBASE-21607
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.3.2
Reporter: Rushabh S Shah
Hbase region server reads failed with following error.
{noformat}
2018-11-30 16:49:18,760 WARN [,queue=12,port=60020] hdfs.BlockReaderFactory -
BlockReaderFactory(fileName=,
block=BP-1618467445--1516873463430:blk_1090719164_16982933): error creating
ShortCircuitReplica.
java.io.IOException: invalid metadata header version 0. Can only handle version
1.
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica.(ShortCircuitReplica.java:129)
at
org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:558)
at
org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:490)
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:782)
at
org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:716)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1087)
at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1444)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1407)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:89)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:834)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1530)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1781)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1624)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:455)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1263)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:297)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:189)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:372)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:220)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2164)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5916)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5890)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2739)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2719)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7197)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7156)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7149)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2250)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:35068)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2373)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
2018-11-30 16:49:18,760 WARN [,queue=12,port=60020]
shortcircuit.ShortCircuitCache - ShortCircuitCache(0x16fd768f): failed to load
1090719164_BP-1618467445--1516873463430
2018-11-30 16:49:18,761 DEBUG [,queue=12,port=60020] ipc.RpcServer -
RpcServer.FifoWFPBQ.default.handler=246,queue=12,port=60020: callId: 46940
service: ClientService methodName: Get size: 443 connection: :48798
deadline: 1543596678759
2018-11-30 16:49:18,761 DEBUG [,queue=12,port=60020] ipc.RpcServer -
RpcServer.FifoWFPBQ.default.handler=246,queue=12,port=60020: callId: 46940
service: ClientService methodName: Get size: 443 connection: :48798
deadline: 1543596678759
java.io.IOException: Could not seek StoreFileScanner[HFileScanner for reader
reader=hdfs://, compression=none, cacheConf=bl
ockCache=LruBlockCache{blockCount=696858, currentSize=7625195376,
freeSize=5689203856, maxSize=13314399232, heapSize=7625195376,
minSize=12648679424, minFactor=0.95, multiSize=6324339712, multiFactor=0.5,
singleSize=3162169856, singleFactor=0.25}, cacheDa
taOnRead=true,
[jira] [Commented] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721714#comment-16721714 ] stack commented on HBASE-21590: --- +1 for branch-2.0 and branch-2.1. Will cut new RCs on w/e. This will be nice to include. > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt, > HBASE-21590.addendum.v1.patch > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21590: -- Fix Version/s: 2.0.4 2.1.2 > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt, > HBASE-21590.addendum.v1.patch > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20917) MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
[
https://issues.apache.org/jira/browse/HBASE-20917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721696#comment-16721696
]
stack commented on HBASE-20917:
---
Go for it [~busbey]
> MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
> --
>
> Key: HBASE-20917
> URL: https://issues.apache.org/jira/browse/HBASE-20917
> Project: HBase
> Issue Type: Bug
> Components: meta, metrics
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0
>
> Attachments: 20917.addendum, 20917.v1.txt, 20917.v2.txt
>
>
> I noticed the following in test output:
> {code}
> 2018-07-21 15:54:43,181 ERROR [RS_CLOSE_REGION-regionserver/172.17.5.4:0-1]
> executor.EventHandler(186): Caught throwable while processing event
> M_RS_CLOSE_REGION
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics.stop(MetaTableMetrics.java:329)
> at
> org.apache.hadoop.hbase.coprocessor.BaseEnvironment.shutdown(BaseEnvironment.java:91)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment.shutdown(RegionCoprocessorHost.java:165)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:290)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.postEnvCall(RegionCoprocessorHost.java:559)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:622)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postClose(RegionCoprocessorHost.java:551)
> at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1678)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1484)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> {code}
> {{requestsMap}} is only initialized for the meta region.
> However, check for meta region is absent in the stop method:
> {code}
> public void stop(CoprocessorEnvironment e) throws IOException {
> // since meta region can move around, clear stale metrics when stop.
> for (String meterName : requestsMap.keySet()) {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20917) MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
[
https://issues.apache.org/jira/browse/HBASE-20917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721669#comment-16721669
]
Sean Busbey commented on HBASE-20917:
-
This is also missing from branch-2.0 and it looks like the same problem is
there. [~stack] unless I hear otherwise from you I'm going to push the same
patch there that I send to branch-2.1.
> MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
> --
>
> Key: HBASE-20917
> URL: https://issues.apache.org/jira/browse/HBASE-20917
> Project: HBase
> Issue Type: Bug
> Components: meta, metrics
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0
>
> Attachments: 20917.addendum, 20917.v1.txt, 20917.v2.txt
>
>
> I noticed the following in test output:
> {code}
> 2018-07-21 15:54:43,181 ERROR [RS_CLOSE_REGION-regionserver/172.17.5.4:0-1]
> executor.EventHandler(186): Caught throwable while processing event
> M_RS_CLOSE_REGION
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics.stop(MetaTableMetrics.java:329)
> at
> org.apache.hadoop.hbase.coprocessor.BaseEnvironment.shutdown(BaseEnvironment.java:91)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment.shutdown(RegionCoprocessorHost.java:165)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:290)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.postEnvCall(RegionCoprocessorHost.java:559)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:622)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postClose(RegionCoprocessorHost.java:551)
> at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1678)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1484)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> {code}
> {{requestsMap}} is only initialized for the meta region.
> However, check for meta region is absent in the stop method:
> {code}
> public void stop(CoprocessorEnvironment e) throws IOException {
> // since meta region can move around, clear stale metrics when stop.
> for (String meterName : requestsMap.keySet()) {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721668#comment-16721668
]
Andrew Purtell commented on HBASE-21520:
Thanks [~openinx], the 1.3.3RC0 binaries are already made and in test. Is this
an essential change?
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.2.10, 1.4.10, 2.1.3, 2.0.5
>
> Attachments: HBASE-21520.branch-1.v4.patch, HBASE-21520.v1.patch,
> HBASE-21520.v2.patch, HBASE-21520.v3.patch, HBASE-21520.v4.patch,
> TestMultiColumnScanner.png, rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20917) MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
[
https://issues.apache.org/jira/browse/HBASE-20917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20917:
Component/s: metrics
meta
> MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
> --
>
> Key: HBASE-20917
> URL: https://issues.apache.org/jira/browse/HBASE-20917
> Project: HBase
> Issue Type: Bug
> Components: meta, metrics
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0
>
> Attachments: 20917.addendum, 20917.v1.txt, 20917.v2.txt
>
>
> I noticed the following in test output:
> {code}
> 2018-07-21 15:54:43,181 ERROR [RS_CLOSE_REGION-regionserver/172.17.5.4:0-1]
> executor.EventHandler(186): Caught throwable while processing event
> M_RS_CLOSE_REGION
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics.stop(MetaTableMetrics.java:329)
> at
> org.apache.hadoop.hbase.coprocessor.BaseEnvironment.shutdown(BaseEnvironment.java:91)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment.shutdown(RegionCoprocessorHost.java:165)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:290)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.postEnvCall(RegionCoprocessorHost.java:559)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:622)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postClose(RegionCoprocessorHost.java:551)
> at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1678)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1484)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> {code}
> {{requestsMap}} is only initialized for the meta region.
> However, check for meta region is absent in the stop method:
> {code}
> public void stop(CoprocessorEnvironment e) throws IOException {
> // since meta region can move around, clear stale metrics when stop.
> for (String meterName : requestsMap.keySet()) {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-20917) MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
[
https://issues.apache.org/jira/browse/HBASE-20917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-20917:
-
reopening so I can backport to branch-2.1 when I'm ready to push HBASE-19722
> MetaTableMetrics#stop references uninitialized requestsMap for non-meta region
> --
>
> Key: HBASE-20917
> URL: https://issues.apache.org/jira/browse/HBASE-20917
> Project: HBase
> Issue Type: Bug
> Components: meta, metrics
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0
>
> Attachments: 20917.addendum, 20917.v1.txt, 20917.v2.txt
>
>
> I noticed the following in test output:
> {code}
> 2018-07-21 15:54:43,181 ERROR [RS_CLOSE_REGION-regionserver/172.17.5.4:0-1]
> executor.EventHandler(186): Caught throwable while processing event
> M_RS_CLOSE_REGION
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics.stop(MetaTableMetrics.java:329)
> at
> org.apache.hadoop.hbase.coprocessor.BaseEnvironment.shutdown(BaseEnvironment.java:91)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment.shutdown(RegionCoprocessorHost.java:165)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:290)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.postEnvCall(RegionCoprocessorHost.java:559)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:622)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postClose(RegionCoprocessorHost.java:551)
> at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1678)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1484)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> {code}
> {{requestsMap}} is only initialized for the meta region.
> However, check for meta region is absent in the stop method:
> {code}
> public void stop(CoprocessorEnvironment e) throws IOException {
> // since meta region can move around, clear stale metrics when stop.
> for (String meterName : requestsMap.keySet()) {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21606) Document use of the meta table load metrics added in HBASE-19722
Sean Busbey created HBASE-21606: --- Summary: Document use of the meta table load metrics added in HBASE-19722 Key: HBASE-21606 URL: https://issues.apache.org/jira/browse/HBASE-21606 Project: HBase Issue Type: Task Components: documentation, meta, metrics, Operability Affects Versions: 2.0.2, 1.4.6, 3.0.0, 1.5.0, 2.2.0, 2.1.3 Reporter: Sean Busbey Assignee: Sean Busbey HBASE-19722 added a great new tool for figuring out where cluster load is coming from. Needs a section in the ref guide * When should I use this? * Why shouldn't I use it all the time? * What does using it look like? * How do I use it? I think all the needed info for making something to answer these questions is in the discussion on HBASE-19722 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-19722:
Attachment: HBASE-19722-branch-2.1.v1.patch
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0, 2.0.2
>
> Attachments: HBASE-19722-branch-2.1.v1.patch,
> HBASE-19722.branch-1.v001.patch, HBASE-19722.branch-1.v002.patch,
> HBASE-19722.master.010.patch, HBASE-19722.master.011.patch,
> HBASE-19722.master.012.patch, HBASE-19722.master.013.patch,
> HBASE-19722.master.014.patch, HBASE-19722.master.015.patch,
> HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-19722:
Status: Patch Available (was: Reopened)
backport for branch-2.1 v1
- backport was clean
- local build + new tests pass
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.0.2, 1.4.6
>
> Attachments: HBASE-19722-branch-2.1.v1.patch,
> HBASE-19722.branch-1.v001.patch, HBASE-19722.branch-1.v002.patch,
> HBASE-19722.master.010.patch, HBASE-19722.master.011.patch,
> HBASE-19722.master.012.patch, HBASE-19722.master.013.patch,
> HBASE-19722.master.014.patch, HBASE-19722.master.015.patch,
> HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721641#comment-16721641
]
Sean Busbey commented on HBASE-19722:
-
read back through the discussion and I'd say the WIP release note isn't
sufficient. Will make a follow-on jira for what I think needs to be in the ref
guide for this. Once that closes I'll add a release note here that references
it.
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0, 2.0.2
>
> Attachments: HBASE-19722.branch-1.v001.patch,
> HBASE-19722.branch-1.v002.patch, HBASE-19722.master.010.patch,
> HBASE-19722.master.011.patch, HBASE-19722.master.012.patch,
> HBASE-19722.master.013.patch, HBASE-19722.master.014.patch,
> HBASE-19722.master.015.patch, HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-19722:
Component/s: Operability
metrics
meta
Coprocessors
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0, 2.0.2
>
> Attachments: HBASE-19722.branch-1.v001.patch,
> HBASE-19722.branch-1.v002.patch, HBASE-19722.master.010.patch,
> HBASE-19722.master.011.patch, HBASE-19722.master.012.patch,
> HBASE-19722.master.013.patch, HBASE-19722.master.014.patch,
> HBASE-19722.master.015.patch, HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-19722:
Priority: Critical (was: Major)
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0, 2.0.2
>
> Attachments: HBASE-19722.branch-1.v001.patch,
> HBASE-19722.branch-1.v002.patch, HBASE-19722.master.010.patch,
> HBASE-19722.master.011.patch, HBASE-19722.master.012.patch,
> HBASE-19722.master.013.patch, HBASE-19722.master.014.patch,
> HBASE-19722.master.015.patch, HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-19722:
Fix Version/s: (was: 2.1.0)
2.2.0
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.2.0, 2.0.2
>
> Attachments: HBASE-19722.branch-1.v001.patch,
> HBASE-19722.branch-1.v002.patch, HBASE-19722.master.010.patch,
> HBASE-19722.master.011.patch, HBASE-19722.master.012.patch,
> HBASE-19722.master.013.patch, HBASE-19722.master.014.patch,
> HBASE-19722.master.015.patch, HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-19722:
-
this never made it into branch-2.1. looks like it landed around the time
branching was happening. reopening while I get it into 2.1 releases.
is the current WIP release note still good? would like to make sure this makes
into the release notes.
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.6, 2.0.2
>
> Attachments: HBASE-19722.branch-1.v001.patch,
> HBASE-19722.branch-1.v002.patch, HBASE-19722.master.010.patch,
> HBASE-19722.master.011.patch, HBASE-19722.master.012.patch,
> HBASE-19722.master.013.patch, HBASE-19722.master.014.patch,
> HBASE-19722.master.015.patch, HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
>
>
>
>
> ===
> *Release Note* (WIP)
> *1. Usage:*
> Use this coprocessor by adding below section to hbase-site.xml
> {{}}
> {{ hbase.coprocessor.region.classes}}
> {{ org.apache.hadoop.hbase.coprocessor.MetaTableMetrics}}
> {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21590: Attachment: HBASE-21590.addendum.v1.patch > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt, > HBASE-21590.addendum.v1.patch > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21590: Status: Patch Available (was: Reopened) addendum v1: - fixes the two long lines called out by checkstyle in precommit > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt, > HBASE-21590.addendum.v1.patch > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-21590: - reopening to attach addendum for the checkstyle fixes I asked for. > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21178) [BC break] : Get and Scan operation with a custom converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21178:
Affects Version/s: 2.1.0
2.0.1
2.0.2
> [BC break] : Get and Scan operation with a custom converter_class not working
> -
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.1.0, 2.0.0, 2.0.1, 2.0.2
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21178.master.001.patch,
> HBASE-21178.master.002.patch, HBASE-21178.master.003.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21178) [BC break] : Get and Scan operation with a custom converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721598#comment-16721598
]
Sean Busbey commented on HBASE-21178:
-
Can we get a release note for this please? It's fixing a critical shortfall
that users upgrading to 2.y need to be aware of.
I've also filed HBASE-21605 for a related documentation change so that folks
considering upgrade can be made aware.
> [BC break] : Get and Scan operation with a custom converter_class not working
> -
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21178.master.001.patch,
> HBASE-21178.master.002.patch, HBASE-21178.master.003.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21605) Need upgrade section note about BC break in 2.0+ fixed by HBASE-21178
Sean Busbey created HBASE-21605: --- Summary: Need upgrade section note about BC break in 2.0+ fixed by HBASE-21178 Key: HBASE-21605 URL: https://issues.apache.org/jira/browse/HBASE-21605 Project: HBase Issue Type: Task Components: documentation Affects Versions: 2.0.2, 2.0.1, 2.0.0, 2.1.0 Reporter: Sean Busbey HBASE-21178 "[BC break] : Get and Scan operation with a custom converter_class not working" fixes how the hbase shell handles custom converters. We need to call out in the upgrade section that folks upgrading from HBase 1.y need to be aware of the change for versions earlier than when HBASE-21178 is included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21565) Delete dead server from dead server list too early leads to concurrent Server Crash Procedures(SCP) for a same server
[
https://issues.apache.org/jira/browse/HBASE-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721528#comment-16721528
]
Hadoop QA commented on HBASE-21565:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
0s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
52s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 34s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}244m 0s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}297m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.util.TestFromClientSide3WoUnsafe |
| | hadoop.hbase.client.TestFromClientSide3 |
| | hadoop.hbase.master.procedure.TestServerCrashProcedureWithReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21565 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951800/HBASE-21565.master.006.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1284c2d4632b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1b08ba7385 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/15286/artifact/patchprocess/whitespace-eol.txt
|
|
[jira] [Commented] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721444#comment-16721444
]
Hadoop QA commented on HBASE-21520:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 27m
55s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} branch-1 passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 3 new + 5 unchanged
- 4 fixed = 8 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 39s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}169m 21s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}216m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterBalanceThrottling |
| | hadoop.hbase.coprocessor.TestMetaTableMetrics |
| | hadoop.hbase.client.TestAdmin2 |
| | hadoop.hbase.master.normalizer.TestSimpleRegionNormalizerOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21520 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-21019) Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721410#comment-16721410
]
Hadoop QA commented on HBASE-21019:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 26s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}259m 29s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}296m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestSnapshotTemporaryDirectory |
| | hadoop.hbase.client.TestFromClientSide3 |
| | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas |
| | hadoop.hbase.client.TestFromClientSide |
| | hadoop.hbase.client.TestFromClientSideWithCoprocessor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21019 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951788/HBASE-21019.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 65a9eeaa0852 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3ff274e22e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
|
[jira] [Commented] (HBASE-21512) Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
[
https://issues.apache.org/jira/browse/HBASE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721408#comment-16721408
]
Hudson commented on HBASE-21512:
Results for branch HBASE-21512
[build #17 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/17/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/17//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/17//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/17//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
> --
>
> Key: HBASE-21512
> URL: https://issues.apache.org/jira/browse/HBASE-21512
> Project: HBase
> Issue Type: Umbrella
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
>
> At least for the RSProcedureDispatcher, with CompletableFuture we do not need
> to set a delay and use a thread pool any more, which could reduce the
> resource usage and also the latency.
> Once this is done, I think we can remove the ClusterConnection completely,
> and start to rewrite the old sync client based on the async client, which
> could reduce the code base a lot for our client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21201) Support to run VerifyReplication MR tool without peerid
[
https://issues.apache.org/jira/browse/HBASE-21201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-21201:
-
Issue Type: Improvement (was: Brainstorming)
> Support to run VerifyReplication MR tool without peerid
> ---
>
> Key: HBASE-21201
> URL: https://issues.apache.org/jira/browse/HBASE-21201
> Project: HBase
> Issue Type: Improvement
> Components: hbase-operator-tools
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Sujit P
>Assignee: Toshihiro Suzuki
>Priority: Major
>
> In some use cases, hbase clients writes to separate clusters(probably
> different datacenters) tables for redundancy. As an administrator/application
> architect, I would like to find out if both cluster tables are in the same
> state (cell by cell). One of the tools that is readily available to use is
> VerifyRep which is part of replication.
> However, it requires peerId to be setup on atleast of the involved cluster.
> PeerId is unnecessary in this use-case scenario and possibly cause unintended
> consequences as the clusters aren't really replication peers neither do We
> prefer them to be.
> Looking at the code:
> Tool attempts to get only the clusterKey which is essentially ZooKeeper
> quorum url
>
> {code:java}
> //VerifyReplication.java
> private static Pair
> getPeerQuorumConfig(final Configuration conf, String peerId)
> .
> .
> return Pair.newPair(peerConfig,
> ReplicationUtils.getPeerClusterConfiguration(peerConfig, conf));
> //ReplicationUtils.java
> public static Configuration getPeerClusterConfiguration(ReplicationPeerConfig
> peerConfig, Configuration baseConf) throws ReplicationException {
> Configuration otherConf;
> try {
> otherConf = HBaseConfiguration.createClusterConf(baseConf,
> peerConfig.getClusterKey());{code}
>
>
> So I would like to propose to update the tool to pass the remote cluster
> ZkQuorum as an argument (ex. --peerQuorumAddress
> clusterBzk1,clusterBzk2,clusterBzk3:2181/hbase-secure ) and use it
> effectively without dependence on replication peerId, similar to
> peerFSAddress. The are certain advantages in doing so as follows:
> * Reduce the development/maintenance of separate tool for above scenario
> * Allow the tool to be more useful for other scenarios as well such as
> ** validating backups in remote cluster HBASE-19106
> ** compare cloned tableA and original tableA in same/remote cluster incase
> of user error before restoring snapshot to original table to find the records
> that need to be added/invalid/missing etc
> ** Allow backup operators who are non-Hbase admins(who shouldn't be adding
> the peerId) to run the tool, since currently only Hbase superuser can add a
> peerId for reasons discussed in HBASE-21163.
> Please post your comments
> Thanks
> cc: [~clayb], [~brfrn169] , [~vrodionov] , [~rashidaligee]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21201) Support to run VerifyReplication MR tool without peerid
[
https://issues.apache.org/jira/browse/HBASE-21201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721405#comment-16721405
]
Toshihiro Suzuki commented on HBASE-21201:
--
I will work on this improvement.
> Support to run VerifyReplication MR tool without peerid
> ---
>
> Key: HBASE-21201
> URL: https://issues.apache.org/jira/browse/HBASE-21201
> Project: HBase
> Issue Type: Brainstorming
> Components: hbase-operator-tools
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Sujit P
>Assignee: Toshihiro Suzuki
>Priority: Major
>
> In some use cases, hbase clients writes to separate clusters(probably
> different datacenters) tables for redundancy. As an administrator/application
> architect, I would like to find out if both cluster tables are in the same
> state (cell by cell). One of the tools that is readily available to use is
> VerifyRep which is part of replication.
> However, it requires peerId to be setup on atleast of the involved cluster.
> PeerId is unnecessary in this use-case scenario and possibly cause unintended
> consequences as the clusters aren't really replication peers neither do We
> prefer them to be.
> Looking at the code:
> Tool attempts to get only the clusterKey which is essentially ZooKeeper
> quorum url
>
> {code:java}
> //VerifyReplication.java
> private static Pair
> getPeerQuorumConfig(final Configuration conf, String peerId)
> .
> .
> return Pair.newPair(peerConfig,
> ReplicationUtils.getPeerClusterConfiguration(peerConfig, conf));
> //ReplicationUtils.java
> public static Configuration getPeerClusterConfiguration(ReplicationPeerConfig
> peerConfig, Configuration baseConf) throws ReplicationException {
> Configuration otherConf;
> try {
> otherConf = HBaseConfiguration.createClusterConf(baseConf,
> peerConfig.getClusterKey());{code}
>
>
> So I would like to propose to update the tool to pass the remote cluster
> ZkQuorum as an argument (ex. --peerQuorumAddress
> clusterBzk1,clusterBzk2,clusterBzk3:2181/hbase-secure ) and use it
> effectively without dependence on replication peerId, similar to
> peerFSAddress. The are certain advantages in doing so as follows:
> * Reduce the development/maintenance of separate tool for above scenario
> * Allow the tool to be more useful for other scenarios as well such as
> ** validating backups in remote cluster HBASE-19106
> ** compare cloned tableA and original tableA in same/remote cluster incase
> of user error before restoring snapshot to original table to find the records
> that need to be added/invalid/missing etc
> ** Allow backup operators who are non-Hbase admins(who shouldn't be adding
> the peerId) to run the tool, since currently only Hbase superuser can add a
> peerId for reasons discussed in HBASE-21163.
> Please post your comments
> Thanks
> cc: [~clayb], [~brfrn169] , [~vrodionov] , [~rashidaligee]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21514) Refactor CacheConfig
[
https://issues.apache.org/jira/browse/HBASE-21514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721389#comment-16721389
]
Hadoop QA commented on HBASE-21514:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 37 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 54s{color}
| {color:red} hbase-server generated 4 new + 184 unchanged - 4 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} hbase-server: The patch generated 0 new + 1040
unchanged - 61 fixed = 1040 total (was 1101) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
2s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 50s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}231m 56s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}273m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.TestSyncReplicationStandbyKillRS |
| | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
| | hadoop.hbase.master.replication.TestRegisterPeerWorkerWhenRestarting |
| | hadoop.hbase.regionserver.TestRecoveredEdits |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21514 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951787/HBASE-21514.master.013.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux efbef3faec7c 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3ff274e22e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs |
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721393#comment-16721393
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #59 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/59/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/59//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/59//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/59//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[
https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721337#comment-16721337
]
Hudson commented on HBASE-21590:
Results for branch master
[build #662 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/662/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/662//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/662//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/662//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Optimize trySkipToNextColumn in StoreScanner a bit
> --
>
> Key: HBASE-21590
> URL: https://issues.apache.org/jira/browse/HBASE-21590
> Project: HBase
> Issue Type: Improvement
> Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10
>
> Attachments: 21590-1.5.txt, HBASE-21590-master.txt
>
>
> See latest comment on HBASE-17958
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21201) Support to run VerifyReplication MR tool without peerid
[
https://issues.apache.org/jira/browse/HBASE-21201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki reassigned HBASE-21201:
Assignee: Toshihiro Suzuki
> Support to run VerifyReplication MR tool without peerid
> ---
>
> Key: HBASE-21201
> URL: https://issues.apache.org/jira/browse/HBASE-21201
> Project: HBase
> Issue Type: Brainstorming
> Components: hbase-operator-tools
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Sujit P
>Assignee: Toshihiro Suzuki
>Priority: Major
>
> In some use cases, hbase clients writes to separate clusters(probably
> different datacenters) tables for redundancy. As an administrator/application
> architect, I would like to find out if both cluster tables are in the same
> state (cell by cell). One of the tools that is readily available to use is
> VerifyRep which is part of replication.
> However, it requires peerId to be setup on atleast of the involved cluster.
> PeerId is unnecessary in this use-case scenario and possibly cause unintended
> consequences as the clusters aren't really replication peers neither do We
> prefer them to be.
> Looking at the code:
> Tool attempts to get only the clusterKey which is essentially ZooKeeper
> quorum url
>
> {code:java}
> //VerifyReplication.java
> private static Pair
> getPeerQuorumConfig(final Configuration conf, String peerId)
> .
> .
> return Pair.newPair(peerConfig,
> ReplicationUtils.getPeerClusterConfiguration(peerConfig, conf));
> //ReplicationUtils.java
> public static Configuration getPeerClusterConfiguration(ReplicationPeerConfig
> peerConfig, Configuration baseConf) throws ReplicationException {
> Configuration otherConf;
> try {
> otherConf = HBaseConfiguration.createClusterConf(baseConf,
> peerConfig.getClusterKey());{code}
>
>
> So I would like to propose to update the tool to pass the remote cluster
> ZkQuorum as an argument (ex. --peerQuorumAddress
> clusterBzk1,clusterBzk2,clusterBzk3:2181/hbase-secure ) and use it
> effectively without dependence on replication peerId, similar to
> peerFSAddress. The are certain advantages in doing so as follows:
> * Reduce the development/maintenance of separate tool for above scenario
> * Allow the tool to be more useful for other scenarios as well such as
> ** validating backups in remote cluster HBASE-19106
> ** compare cloned tableA and original tableA in same/remote cluster incase
> of user error before restoring snapshot to original table to find the records
> that need to be added/invalid/missing etc
> ** Allow backup operators who are non-Hbase admins(who shouldn't be adding
> the peerId) to run the tool, since currently only Hbase superuser can add a
> peerId for reasons discussed in HBASE-21163.
> Please post your comments
> Thanks
> cc: [~clayb], [~brfrn169] , [~vrodionov] , [~rashidaligee]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21575) memstore above high watermark message is logged too much
[
https://issues.apache.org/jira/browse/HBASE-21575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721338#comment-16721338
]
Hudson commented on HBASE-21575:
Results for branch master
[build #662 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/662/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/662//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/662//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/662//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> memstore above high watermark message is logged too much
>
>
> Key: HBASE-21575
> URL: https://issues.apache.org/jira/browse/HBASE-21575
> Project: HBase
> Issue Type: Bug
> Components: logging, regionserver
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21575.01.patch, HBASE-21575.patch
>
>
> 100s of Mb of logs like this, in a tight loop:
> {noformat}
> 2018-12-08 10:27:00,462 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3646ms
> 2018-12-08 10:27:00,463 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3647ms
> 2018-12-08 10:27:00,463 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3647ms
> 2018-12-08 10:27:00,464 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3648ms
> 2018-12-08 10:27:00,464 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3648ms
> 2018-12-08 10:27:00,465 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3649ms
> 2018-12-08 10:27:00,465 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3649ms
> 2018-12-08 10:27:00,466 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3650ms
> 2018-12-08 10:27:00,466 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3650ms
> 2018-12-08 10:27:00,467 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3651ms
> 2018-12-08 10:27:00,469 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3653ms
> 2018-12-08 10:27:00,470 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3654ms
> 2018-12-08 10:27:00,470 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3654ms
> 2018-12-08 10:27:00,471 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3655ms
> 2018-12-08 10:27:00,471 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3655ms
> 2018-12-08 10:27:00,472 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3656ms
> 2018-12-08 10:27:00,472 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3656ms
> 2018-12-08 10:27:00,473 WARN
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 3657ms
>
[jira] [Updated] (HBASE-21592) quota.addGetResult(r) throw NPE
[
https://issues.apache.org/jira/browse/HBASE-21592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xuqinya updated HBASE-21592:
Affects Version/s: 2.0.2
1.2.9
> quota.addGetResult(r) throw NPE
> -
>
> Key: HBASE-21592
> URL: https://issues.apache.org/jira/browse/HBASE-21592
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.2, 1.2.9
>Reporter: xuqinya
>Assignee: xuqinya
>Priority: Major
> Attachments: HBASE-21592.master.0001.patch
>
>
> Setting the RPC quota, table.exists(Get) cause quota.addGetResult(r) throw
> NPE.
>
> set_quota TYPE => THROTTLE, NAMESPACE => 'ns1', LIMIT => '1000req/sec'
> Connection conn = ConnectionFactory.createConnection(config);
> Table htable = conn.getTable(TableName.valueOf("ns1:t1"));
> boolean exists = htable.exists(new Get(Bytes.toBytes("123")));
> log:
> java.io.IOException: java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2183)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.quotas.QuotaUtil.calculateResultSize(QuotaUtil.java:282)
> at
> org.apache.hadoop.hbase.quotas.DefaultOperationQuota.addGetResult(DefaultOperationQuota.java:99)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1907)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2135)
> ... 4 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[
https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721289#comment-16721289
]
Hudson commented on HBASE-21590:
Results for branch branch-2
[build #1557 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1557/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1557//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1557//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1557//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Optimize trySkipToNextColumn in StoreScanner a bit
> --
>
> Key: HBASE-21590
> URL: https://issues.apache.org/jira/browse/HBASE-21590
> Project: HBase
> Issue Type: Improvement
> Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10
>
> Attachments: 21590-1.5.txt, HBASE-21590-master.txt
>
>
> See latest comment on HBASE-17958
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721204#comment-16721204
]
Zheng Hu commented on HBASE-21520:
--
Patch for branch-1, waiting for hadoop QA. [~apurtell], Do the 1.3.3 release
need this patch ? Thanks.
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.2.10, 1.4.10, 2.1.3, 2.0.5
>
> Attachments: HBASE-21520.branch-1.v4.patch, HBASE-21520.v1.patch,
> HBASE-21520.v2.patch, HBASE-21520.v3.patch, HBASE-21520.v4.patch,
> TestMultiColumnScanner.png, rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21565) Delete dead server from dead server list too early leads to concurrent Server Crash Procedures(SCP) for a same server
[
https://issues.apache.org/jira/browse/HBASE-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingyun Tian updated HBASE-21565:
-
Attachment: HBASE-21565.master.006.patch
> Delete dead server from dead server list too early leads to concurrent Server
> Crash Procedures(SCP) for a same server
> -
>
> Key: HBASE-21565
> URL: https://issues.apache.org/jira/browse/HBASE-21565
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Critical
> Attachments: HBASE-21565.master.001.patch,
> HBASE-21565.master.002.patch, HBASE-21565.master.003.patch,
> HBASE-21565.master.004.patch, HBASE-21565.master.005.patch,
> HBASE-21565.master.006.patch
>
>
> There are 2 kinds of SCP for a same server will be scheduled during cluster
> restart, one is ZK session timeout, the other one is new server report in
> will cause the stale one do fail over. The only barrier for these 2 kinds of
> SCP is check if the server is in the dead server list.
> {code}
> if (this.deadservers.isDeadServer(serverName)) {
> LOG.warn("Expiration called on {} but crash processing already in
> progress", serverName);
> return false;
> }
> {code}
> But the problem is when master finish initialization, it will delete all
> stale servers from dead server list. Thus when the SCP for ZK session timeout
> come in, the barrier is already removed.
> Here is the logs that how this problem occur.
> {code}
> 2018-12-07,11:42:37,589 INFO
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: Start pid=9,
> state=RUNNABLE:SERVER_CRASH_START, hasLock=true; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> 2018-12-07,11:42:58,007 INFO
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: Start pid=444,
> state=RUNNABLE:SERVER_CRASH_START, hasLock=true; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> {code}
> Now we can see two SCP are scheduled for the same server.
> But the first procedure is finished after the second SCP starts.
> {code}
> 2018-12-07,11:43:08,038 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=9,
> state=SUCCESS, hasLock=false; ServerCrashProcedure
> server=c4-hadoop-tst-st27.bj,29100,1544153846859, splitWal=true, meta=false
> in 30.5340sec
> {code}
> Thus it will leads the problem that regions will be assigned twice.
> {code}
> 2018-12-07,12:16:33,039 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: rit=OPEN,
> location=c4-hadoop-tst-st28.bj,29100,1544154149607, table=test_failover,
> region=459b3130b40caf3b8f3e1421766f4089 reported OPEN on
> server=c4-hadoop-tst-st29.bj,29100,1544154149615 but state has otherwise
> {code}
> And here we can see the server is removed from dead server list before the
> second SCP starts.
> {code}
> 2018-12-07,11:42:44,938 DEBUG org.apache.hadoop.hbase.master.DeadServer:
> Removed c4-hadoop-tst-st27.bj,29100,1544153846859 ; numProcessing=3
> {code}
> Thus we should not delete dead server from dead server list immediately.
> Patch to fix this problem will be upload later.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[
https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721211#comment-16721211
]
Hudson commented on HBASE-21590:
Results for branch branch-1
[build #591 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/591/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/591//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/591//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/591//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Optimize trySkipToNextColumn in StoreScanner a bit
> --
>
> Key: HBASE-21590
> URL: https://issues.apache.org/jira/browse/HBASE-21590
> Project: HBase
> Issue Type: Improvement
> Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10
>
> Attachments: 21590-1.5.txt, HBASE-21590-master.txt
>
>
> See latest comment on HBASE-17958
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721210#comment-16721210
]
Guanghao Zhang commented on HBASE-21578:
Pushed to branch-2 and master. Thanks [~Yi Mei] for contributing.
> Fix wrong throttling exception for capacity unit
>
>
> Key: HBASE-21578
> URL: https://issues.apache.org/jira/browse/HBASE-21578
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21578.master.001.patch,
> HBASE-21578.master.002.patch
>
>
> HBASE-21034 provides a new throttle type: capacity unit, but the throttling
> exception is confusing:
>
> {noformat}
> 2018-12-11 14:38:41,503 DEBUG [Time-limited test]
> client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
> started=0 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write size limit
> exceeded - wait 10sec
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:106)
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteSizeExceeded(RpcThrottlingException.java:96)
> at
> org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:179){noformat}
> Need to make the exception more clearly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-21578:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
3.0.0
Status: Resolved (was: Patch Available)
> Fix wrong throttling exception for capacity unit
>
>
> Key: HBASE-21578
> URL: https://issues.apache.org/jira/browse/HBASE-21578
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21578.master.001.patch,
> HBASE-21578.master.002.patch
>
>
> HBASE-21034 provides a new throttle type: capacity unit, but the throttling
> exception is confusing:
>
> {noformat}
> 2018-12-11 14:38:41,503 DEBUG [Time-limited test]
> client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
> started=0 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write size limit
> exceeded - wait 10sec
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:106)
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteSizeExceeded(RpcThrottlingException.java:96)
> at
> org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:179){noformat}
> Need to make the exception more clearly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21520:
-
Fix Version/s: 1.4.10
1.2.10
1.5.0
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.2.10, 1.4.10, 2.1.3, 2.0.5
>
> Attachments: HBASE-21520.branch-1.v4.patch, HBASE-21520.v1.patch,
> HBASE-21520.v2.patch, HBASE-21520.v3.patch, HBASE-21520.v4.patch,
> TestMultiColumnScanner.png, rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21520:
-
Attachment: HBASE-21520.branch-1.v4.patch
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21520.branch-1.v4.patch, HBASE-21520.v1.patch,
> HBASE-21520.v2.patch, HBASE-21520.v3.patch, HBASE-21520.v4.patch,
> TestMultiColumnScanner.png, rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20523) PE tool should support configuring client side buffering sizes
[
https://issues.apache.org/jira/browse/HBASE-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721160#comment-16721160
]
Hudson commented on HBASE-20523:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> PE tool should support configuring client side buffering sizes
> --
>
> Key: HBASE-20523
> URL: https://issues.apache.org/jira/browse/HBASE-20523
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.1.0, 1.5.0, 2.0.1, 1.4.5, 1.2.7
>
> Attachments: HBASE-20523.patch, HBASE-20523_1.patch,
> HBASE-20523_branch-1.2.patch, HBASE-20523_branch-1.4.patch,
> HBASE-20523_branch-1.patch
>
>
> The client side buffering size impacts the write load and the write
> performance. Hence for testing purpose it is better we allow client side
> buffering to be configurable in PE. Already YCSB has such facility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-16910) Avoid NPE when starting StochasticLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-16910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721121#comment-16721121
]
Hudson commented on HBASE-16910:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Avoid NPE when starting StochasticLoadBalancer
> --
>
> Key: HBASE-16910
> URL: https://issues.apache.org/jira/browse/HBASE-16910
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 1.4.0, 1.3.3, 2.0.0
>
> Attachments: HBASE-16910-branch-1.patch, HBASE-16910-v1.patch,
> HBASE-16910.patch
>
>
> When master start, it initialize StochasticLoadBalancer.
> {code}
> this.balancer.setClusterStatus(getClusterStatus());
> this.balancer.setMasterServices(this);
> {code}
> It first setClusterStatus(), then setMasterService(). But in setClusterStatus
> method, it use master service which is not initialized. So it will throw NPE.
> {code}
> int tablesCount = isByTable ? services.getTableDescriptors().getAll().size()
> : 1;
> {code}
> It happens when set isByTable is ture.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20141) Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
[
https://issues.apache.org/jira/browse/HBASE-20141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721154#comment-16721154
]
Hudson commented on HBASE-20141:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
> --
>
> Key: HBASE-20141
> URL: https://issues.apache.org/jira/browse/HBASE-20141
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.0, 2.0.0-beta-1, 1.4.2
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.3, 2.0.0
>
> Attachments: HBASE-20141.master.001.patch,
> HBASE-20141.master.002.patch, HBASE-20141.master.003.patch,
> HBASE-20141.master.004.patch
>
>
> HBASE-19435 implements a fix for reopening file channels when they are
> unnexpected closed
> to avoid disabling the BucketCache. However, it was missed that the the
> channels might not
> actually be completely closed (the write or read channel might still be open
> (see
> https://docs.oracle.com/javase/7/docs/api/java/nio/channels/ClosedChannelException.html)
> This commit closes any open channels before creating a new channel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20557) Backport HBASE-17215 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721168#comment-16721168
]
Hudson commented on HBASE-20557:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Backport HBASE-17215 to branch-1
>
>
> Key: HBASE-20557
> URL: https://issues.apache.org/jira/browse/HBASE-20557
> Project: HBase
> Issue Type: Sub-task
> Components: HFile, master
>Affects Versions: 1.4.4, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20557.branch-1.001.patch,
> HBASE-20557.branch-1.002.patch, HBASE-20557.branch-1.003.patch
>
>
> As part of HBASE-20555, HBASE-17215 is the second patch that is needed for
> backporting HBASE-18083
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal usage
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721124#comment-16721124
]
Hudson commented on HBASE-17072:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal
> usage
>
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 1.2.0, 2.0.0
>Reporter: Eiichi Sato
>Assignee: Eiichi Sato
>Priority: Critical
> Fix For: 1.4.0, 0.98.24, 1.3.3, 2.0.0
>
> Attachments: HBASE-17072-0.98.patch, HBASE-17072.branch-1.001.patch,
> HBASE-17072.master.001.patch, HBASE-17072.master.002.patch,
> HBASE-17072.master.003.patch, HBASE-17072.master.004.patch,
> HBASE-17072.master.005.patch, HBASE-17072.master.005.patch,
> disable-block-header-cache.patch, mat-threadlocals.png, mat-threads.png,
> metrics.png, slave1.svg, slave2.svg, slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
[jira] [Commented] (HBASE-20808) Wrong shutdown order between Chores and ChoreService
[
https://issues.apache.org/jira/browse/HBASE-20808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721167#comment-16721167
]
Hudson commented on HBASE-20808:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Wrong shutdown order between Chores and ChoreService
>
>
> Key: HBASE-20808
> URL: https://issues.apache.org/jira/browse/HBASE-20808
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Nihal Jain
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20808.branch-1.001.patch,
> HBASE-20808.branch-1.002.patch, HBASE-20808.master.001.patch,
> HBASE-20808.master.002.patch, HBASE-20808.master.003.patch,
> HBASE-20808.master.004.patch, HBASE-20808.master.addendum.patch
>
>
> When stopping master, {{ChoreService}}, which serves all the chores, is
> stopped before canceling all running chores.
> It should cancel all running chores, then shutdown {{ChoreService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21582) If call HBaseAdmin#snapshotAsync but forget call isSnapshotFinished, then SnapshotHFileCleaner will skip to run every time
[
https://issues.apache.org/jira/browse/HBASE-21582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721189#comment-16721189
]
Hudson commented on HBASE-21582:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> If call HBaseAdmin#snapshotAsync but forget call isSnapshotFinished, then
> SnapshotHFileCleaner will skip to run every time
> --
>
> Key: HBASE-21582
> URL: https://issues.apache.org/jira/browse/HBASE-21582
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.2, 1.2.10, 1.4.10, 2.0.5
>
> Attachments: HBASE-21582.branch-1.v3.patch, HBASE-21582.v1.patch,
> HBASE-21582.v2.patch, HBASE-21582.v3.patch
>
>
> This is because we remove the SnapshotSentinel from snapshotHandlers in
> SnapshotManager#cleanupSentinels. Only when the following 3 case, the
> cleanupSentinels will be called:
> 1. SnapshotManager#isSnapshotDone;
> 2. SnapshotManager#takeSnapshot;
> 3. SnapshotManager#restoreOrCloneSnapshot
> So if no isSnapshotDone called, or no further snapshot taking, or snapshot
> restore/clone. the SnapshotSentinel will always be keep in snapshotHandlers.
> But after HBASE-21387, Only when no snapshot taking, the
> SnapshotHFileCleaner will check the unref files and clean.
> I found this bug, because in our XiaoMi branch-2, we implement the soft
> delete feature, which means if someone delete a table, then master will
> create a snapshot firstly, after that, the table deletion begain. the
> implementation is quite simple, we use the snapshotManager to create a
> snapshot.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> index 8f42e4a..6da6a64 100644
> --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> @@ -2385,12 +2385,6 @@ public class HMaster extends HRegionServer implements
> MasterServices {
>protected void run() throws IOException {
> getMaster().getMasterCoprocessorHost().preDeleteTable(tableName);
>
> +if (snapshotBeforeDelete) {
> + LOG.info("Take snaposhot for " + tableName + " before deleting");
> + snapshotManager
> +
> .takeSnapshot(SnapshotDescriptionUtils.getSnapshotNameForDeletedTable(tableName));
> +}
> +
> LOG.info(getClientIdAuditPrefix() + " delete " + tableName);
>
> // TODO: We can handle/merge duplicate request
> {code}
> In the master, I found the endless log after delete a table:
> {code}
> org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache: Not checking
> unreferenced files since snapshot is running, it will skip to clean the
> HFiles this time
> {code}
> This is because the snapshotHandlers never be cleaned after call the
> snapshotManager#takeSnapshot. I think the asynSnapshot may has the same
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[
https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721173#comment-16721173
]
Hudson commented on HBASE-18083:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Make large/small file clean thread number configurable in HFileCleaner
> --
>
> Key: HBASE-18083
> URL: https://issues.apache.org/jira/browse/HBASE-18083
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-18083.patch, HBASE-18083.v2.patch,
> HBASE-18083.v3.patch
>
>
> Currently we have only one thread for both large and small file cleaning, but
> when write pressure is huge we might need more cleaner threads, so we need to
> make the thread number configurable.
> We observed more than 1.8PB data in archive directory online due to business
> access rate change, and this proposal is one of the necessary changes
> required.
> Default value of the configurations would still be left to 1 to keep low
> pressure to NN for normal case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21275) Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http server (branch 1 only)
[
https://issues.apache.org/jira/browse/HBASE-21275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721187#comment-16721187
]
Hudson commented on HBASE-21275:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http
> server (branch 1 only)
>
>
> Key: HBASE-21275
> URL: https://issues.apache.org/jira/browse/HBASE-21275
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 1.5.0, 1.3.3, 1.4.9
>
> Attachments: HBASE-21275-branch-1.001.patch,
> HBASE-21275-branch-1.2.001.patch, HBASE-21275-branch-1.2.002.patch,
> HBASE-21275-branch-1.2.003.patch, HBASE-21275-branch-1.2.003.patch,
> HBASE-21275-branch-1.4.001.patch
>
>
> There's been a reasonable number of users running thrift http server on hbase
> 1.x suffering with security audit tests pointing thrift server allows TRACE
> requests.
> After doing some search, I can see HBASE-20406 added restrictions for
> TRACE/OPTIONS method when Thrift is running over http, but it relies on many
> other commits applied to thrift http server. This patch was later reverted
> from master. Then again later, HBASE-20004 had made TRACE/OPTIONS
> configurable via "*hbase.thrift.http.allow.options.method*" property, with
> both methods being disabled by default. This also seems to rely on many
> changes applied to thrift http server, and a branch 1 compatible patch does
> not seem feasible.
> A solution for branch 1 is pretty simple though, am proposing a patch that
> simply uses *WebAppContext*, instead of *Context*, as the context for the
> *HttpServer* instance. *WebAppContext* will already restrict TRACE methods by
> default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19458) Allow building HBase 1.3.x against Hadoop 2.8
[
https://issues.apache.org/jira/browse/HBASE-19458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721191#comment-16721191
]
Hudson commented on HBASE-19458:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Allow building HBase 1.3.x against Hadoop 2.8
> -
>
> Key: HBASE-19458
> URL: https://issues.apache.org/jira/browse/HBASE-19458
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Major
> Fix For: 1.3.3
>
> Attachments: 19458.txt
>
>
> Currently the 1.3 branch cannot be built against Hadoop 2.8.2. (that latest
> stable 2.x release).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21546) ConnectException in TestThriftHttpServer
[
https://issues.apache.org/jira/browse/HBASE-21546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721188#comment-16721188
]
Hudson commented on HBASE-21546:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> ConnectException in TestThriftHttpServer
>
>
> Key: HBASE-21546
> URL: https://issues.apache.org/jira/browse/HBASE-21546
> Project: HBase
> Issue Type: Bug
> Components: test, Thrift
>Affects Versions: 1.5.0, 1.4.9
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.9
>
> Attachments: HBASE-21546.branch-1.01.patch
>
>
> TestThriftHttpServer is the first on the flaky list for branch-1 and
> branch-1.4 with approximately 60% failure rate.
> Thrift server is not yet accepting request at the time the test starts.
> java.net.ConnectException: Connection refused (Connection refused) at
> org.apache.hadoop.hbase.thrift.TestThriftHttpServer.checkHttpMethods(TestThriftHttpServer.java:275)
> at
> org.apache.hadoop.hbase.thrift.TestThriftHttpServer.testThriftServerHttpOptionsForbiddenWhenOptionsDisabled(TestThriftHttpServer.java:176)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20857) JMX - add Balancer status = enabled / disabled
[
https://issues.apache.org/jira/browse/HBASE-20857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721186#comment-16721186
]
Hudson commented on HBASE-20857:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> JMX - add Balancer status = enabled / disabled
> --
>
> Key: HBASE-20857
> URL: https://issues.apache.org/jira/browse/HBASE-20857
> Project: HBase
> Issue Type: Improvement
> Components: API, master, metrics, REST, tooling, Usability
>Reporter: Hari Sekhon
>Assignee: Kiran Kumar Maturi
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-20857.branch-1.4.001.patch,
> HBASE-20857.branch-1.4.002.patch
>
>
> Add HBase Balancer enabled/disabled status to JMX API on HMaster.
> Right now the HMaster will give a warning near the top of HMaster UI if
> balancer is disabled, but scraping this is for monitoring integration is not
> nice, it should be available in JMX API as there is already a
> Master,sub=Balancer bean with metrics for the balancer ops etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18549) Unclaimed replication queues can go undetected
[
https://issues.apache.org/jira/browse/HBASE-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721182#comment-16721182
]
Hudson commented on HBASE-18549:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Unclaimed replication queues can go undetected
> --
>
> Key: HBASE-18549
> URL: https://issues.apache.org/jira/browse/HBASE-18549
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Ashu Pachauri
>Assignee: Xu Cang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-18549-.master.001.patch,
> HBASE-18549-.master.002.patch, HBASE-18549-.master.003.patch,
> HBASE-18549-.master.004.patch, HBASE-18549.branch-1.001.patch,
> HBASE-18549.branch-1.001.patch
>
>
> We have come across this situation multiple times where a zookeeper issues
> can cause NodeFailoverWorker to fail picking up replication queue for a dead
> region server silently. One example is when the znode size for a particular
> queue exceed jute.maxBuffer value.
> There can be other situations that may lead to this and just go undetected.
> We need to have a metric for number of unclaimed replication queues. This
> will help in mitigating the problem through alerting on the metric and
> identifying underlying issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21359) Fix build problem against Hadoop 2.8.5
[
https://issues.apache.org/jira/browse/HBASE-21359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721190#comment-16721190
]
Hudson commented on HBASE-21359:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Fix build problem against Hadoop 2.8.5
> --
>
> Key: HBASE-21359
> URL: https://issues.apache.org/jira/browse/HBASE-21359
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.8
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 1.5.0, 1.3.3, 1.4.9
>
> Attachments: HBASE-21359-branch-1.patch
>
>
> 1.4.8 build fails against Hadoop 2.8.5. The fix is an easy change to
> supplemental-models.xml.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721181#comment-16721181
]
Hudson commented on HBASE-20734:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Colocate recovered edits directory with hbase.wal.dir
> -
>
> Key: HBASE-20734
> URL: https://issues.apache.org/jira/browse/HBASE-20734
> Project: HBase
> Issue Type: Improvement
> Components: MTTR, Recovery, wal
>Reporter: Ted Yu
>Assignee: Zach York
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-20734.branch-1.001.patch,
> HBASE-20734.branch-1.002.patch, HBASE-20734.branch-1.003.patch,
> HBASE-20734.branch-1.004.patch, HBASE-20734.branch-1.005.patch,
> HBASE-20734.master.001.patch, HBASE-20734.master.002.patch,
> HBASE-20734.master.003.patch, HBASE-20734.master.004.patch,
> HBASE-20734.master.005.patch, HBASE-20734.master.006.patch,
> HBASE-20734.master.007.patch, HBASE-20734.master.008.patch,
> HBASE-20734.master.009.patch, HBASE-20734.master.010.patch,
> HBASE-20734.master.011.patch, HBASE-20734.master.012.patch
>
>
> During investigation of HBASE-20723, I realized that we wouldn't get the best
> performance when hbase.wal.dir is configured to be on different (fast) media
> than hbase rootdir w.r.t. recovered edits since recovered edits directory is
> currently under rootdir.
> Such setup may not result in fast recovery when there is region server
> failover.
> This issue is to find proper (hopefully backward compatible) way in
> colocating recovered edits directory with hbase.wal.dir .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20695) Implement table level RegionServer replication metrics
[
https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721176#comment-16721176
]
Hudson commented on HBASE-20695:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Implement table level RegionServer replication metrics
> ---
>
> Key: HBASE-20695
> URL: https://issues.apache.org/jira/browse/HBASE-20695
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HBASE-20695.master.001.patch,
> HBASE-20695.master.002.patch, HBASE-20695.master.003.patch,
> HBASE-20695.master.004.patch, HBASE-20695.master.005.patch,
> HBASE-20695.master.006.patch, HBASE-20695.master.007.patch,
> HBASE-20695.master.008.patch, HBASE-20695.master.009.patch,
> HBASE-20695.master.010.patch
>
>
> Region server metrics now are mainly global metrics. It would be nice to have
> table level metrics such as table level source.AgeOfLastShippedOp to indicate
> operators which table's replication is lagging behind.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[
https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721174#comment-16721174
]
Hudson commented on HBASE-20401:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
> --
>
> Key: HBASE-20401
> URL: https://issues.apache.org/jira/browse/HBASE-20401
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.4.6, 2.2.0, 2.1.1, 2.0.2
>
> Attachments: HBASE-20401.branch-1.001.patch,
> HBASE-20401.branch-1.002.patch, HBASE-20401.branch-1.003.patch,
> HBASE-20401.master.001.patch, HBASE-20401.master.002.patch,
> HBASE-20401.master.003.patch, HBASE-20401.master.004.patch,
> HBASE-20401.master.005.patch, HBASE-20401.master.006.patch,
> HBASE-20401.master.007.patch
>
>
> When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls
> CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for
> notification (notify) from the fs.delete file thread. there might be two
> situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished
> when LogClearner call getResult.
> # fs.delete never complete (strange but possible), then we need to wait for
> a max of 60 seconds. here, 60 seconds might be too long
> # getResult is waiting in the period of 500 milliseconds, but the fs.delete
> has completed and setFromClear is set but yet notify(). one might want to
> tune this 500 milliseconds to 200 or less .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721178#comment-16721178
]
Hudson commented on HBASE-20723:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.1.0, 1.5.0, 1.3.3, 1.4.6
>
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20616) TruncateTableProcedure is stuck in retry loop in TRUNCATE_TABLE_CREATE_FS_LAYOUT state
[
https://issues.apache.org/jira/browse/HBASE-20616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721163#comment-16721163
]
Hudson commented on HBASE-20616:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> TruncateTableProcedure is stuck in retry loop in
> TRUNCATE_TABLE_CREATE_FS_LAYOUT state
> --
>
> Key: HBASE-20616
> URL: https://issues.apache.org/jira/browse/HBASE-20616
> Project: HBase
> Issue Type: Bug
> Components: amv2
> Environment: HDP-2.5.3
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: 20616.master.004.patch, HBASE-20616.master.001.patch,
> HBASE-20616.master.002.patch, HBASE-20616.master.003.patch,
> HBASE-20616.master.004.patch
>
>
> At first, TruncateTableProcedure failed to write some files to HDFS in
> TRUNCATE_TABLE_CREATE_FS_LAYOUT state for some reason.
> {code:java}
> 2018-05-15 08:00:25,346 WARN [ProcedureExecutorThread-8]
> procedure.TruncateTableProcedure: Retriable error trying to truncate
> table=: state=TRUNCATE_TABLE_CREATE_FS_LAYOUT
> java.io.IOException: java.util.concurrent.ExecutionException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /apps/hbase/data/.tmp/data.regioninfo could
> only be replicated to 0 nodes instead of minReplication (=1). There are number of DNs> datanode(s) running and no node(s) are excluded in this
> operation.
> ...
> {code}
> But at this time, seemed like writing some files to HDFS was successful.
> And then, TruncateTableProcedure was stuck in retry loop in
> TRUNCATE_TABLE_CREATE_FS_LAYOUT state. At this point, the following log
> messages were shown repeatedly in the master log:
> {code:java}
> 2018-05-15 08:00:25,463 WARN [ProcedureExecutorThread-8]
> procedure.TruncateTableProcedure: Retriable error trying to truncate
> table=: state=TRUNCATE_TABLE_CREATE_FS_LAYOUT
> java.io.IOException: java.util.concurrent.ExecutionException:
> java.io.IOException: The specified region already exists on disk:
> hdfs:///apps/hbase/data/.tmp/data///
> ...
> {code}
> It seems like this is because TruncateTableProcedure tried to write the files
> that were written successfully in the first try.
> I think we need to delete all the files and directories that are written
> successfully in the previous try before retrying the
> TRUNCATE_TABLE_CREATE_FS_LAYOUT state.
> Actually, this issue was observed in HDP-2.5.3, but I think the upstream has
> the same issue. Also, it looks to me that CreateTableProcedure has a similar
> issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19391) Calling HRegion#initializeRegionInternals from a region replica can still re-create a region directory
[
https://issues.apache.org/jira/browse/HBASE-19391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721151#comment-16721151
]
Hudson commented on HBASE-19391:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Calling HRegion#initializeRegionInternals from a region replica can still
> re-create a region directory
> --
>
> Key: HBASE-19391
> URL: https://issues.apache.org/jira/browse/HBASE-19391
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4, 1.4.2
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Major
> Fix For: 1.5.0, 2.0.0-beta-2, 1.3.3, 1.4.3, 2.0.0
>
> Attachments:
> 0001-HBASE-19391-Calling-HRegion-initializeRegionInternal.patch,
> HBASE-19391.master.v0.patch
>
>
> This is a follow up from HBASE-18024. There stills a chance that attempting
> to open a region that is not the default region replica can still create a
> GC'd region directory by the CatalogJanitor causing inconsistencies with hbck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721171#comment-16721171
]
Hudson commented on HBASE-17854:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch,
> HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch,
> HBASE-17854.v5.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived)
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs
> from small queue, based on the experience that in real world there'll be much
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete
> from the larger file in the queues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20559) Backport HBASE-18083 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721172#comment-16721172
]
Hudson commented on HBASE-20559:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Backport HBASE-18083 to branch-1
>
>
> Key: HBASE-20559
> URL: https://issues.apache.org/jira/browse/HBASE-20559
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
>Affects Versions: 1.4.4, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20559.branch-1.001.patch,
> HBASE-20559.branch-1.002.patch
>
>
> As part of HBASE-20555, HBASE-18083 is the fourth/last patch that is needed
> for backporting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20052) TestRegionOpen#testNonExistentRegionReplica fails due to NPE
[
https://issues.apache.org/jira/browse/HBASE-20052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721153#comment-16721153
]
Hudson commented on HBASE-20052:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> TestRegionOpen#testNonExistentRegionReplica fails due to NPE
>
>
> Key: HBASE-20052
> URL: https://issues.apache.org/jira/browse/HBASE-20052
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.3, 2.0.0
>
> Attachments: 20052.v1.txt, 20052.v2.txt
>
>
> After HBASE-19391 was integrated, the following test failure can be observed:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.TestRegionOpen.testNonExistentRegionReplica(TestRegionOpen.java:122)
> {code}
> This was due null being returned from
> HRegionFileSystem#createRegionOnFileSystem().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20732) Shutdown scan pool when master is stopped.
[
https://issues.apache.org/jira/browse/HBASE-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721166#comment-16721166
]
Hudson commented on HBASE-20732:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Shutdown scan pool when master is stopped.
> --
>
> Key: HBASE-20732
> URL: https://issues.apache.org/jira/browse/HBASE-20732
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20732.branch-1.001.patch,
> HBASE-20732.master.001.patch, HBASE-20732.master.002.patch,
> HBASE-20732.master.003.patch, HBASE-20732.master.004.patch,
> HBASE-20732.master.005.patch, HBASE-20732.master.006.patch,
> HBASE-20732.master.007.patch, HBASE-20732.master.008.patch,
> HBASE-20732.master.009.patch
>
>
> If master is stopped, {{DirScanPool}} is kept open. This is found by
> [~chia7712] when reviewing HBASE-20352.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[
https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721177#comment-16721177
]
Hudson commented on HBASE-17437:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Support specifying a WAL directory outside of the root directory
>
>
> Key: HBASE-17437
> URL: https://issues.apache.org/jira/browse/HBASE-17437
> Project: HBase
> Issue Type: Improvement
> Components: Filesystem Integration, wal
>Affects Versions: 1.2.4
>Reporter: Yishan Yang
>Assignee: Zach York
>Priority: Major
> Labels: patch
> Fix For: 1.4.0, 1.3.3, 2.0.0
>
> Attachments: HBASE-17437.branch-1.001.patch,
> HBASE-17437.branch-1.002.patch, HBASE-17437.branch-1.003.patch,
> HBASE-17437.branch-1.004.patch, HBASE-17437.master.001.patch,
> HBASE-17437.master.002.patch, HBASE-17437.master.003.patch,
> HBASE-17437.master.004.patch, HBASE-17437.master.005.patch,
> HBASE-17437.master.006.patch, HBASE-17437.master.007.patch,
> HBASE-17437.master.008.patch, HBASE-17437.master.009.patch,
> HBASE-17437.master.010.patch, HBASE-17437.master.011.patch,
> HBASE-17437.master.012.patch, hbase-17437-branch-1.2.patch,
> hbase-17437-master.patch
>
>
> Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some
> FileSystems (such as Amazon S3) don’t support append or consistent writes.
> These two properties are imperative for the WAL in order to avoid loss of
> writes. However, StoreFiles don’t necessarily need the same consistency
> guarantees (since writes are cached locally and if writes fail, they can
> always be replayed from the WAL).
>
> This JIRA aims to allow users to configure a log directory (for WALs) that is
> outside of the root directory or even in a different FileSystem. The default
> value will still put the log directory under the root directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20352) [Chore] Backport HBASE-18309 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721156#comment-16721156
]
Hudson commented on HBASE-20352:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [Chore] Backport HBASE-18309 to branch-1
>
>
> Key: HBASE-20352
> URL: https://issues.apache.org/jira/browse/HBASE-20352
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.4
>
> Attachments: HBASE-20352.branch-1.001.patch
>
>
> Using multiple threads to scan directory and to clean old WALs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20672) New metrics ReadRequestRate and WriteRequestRate
[
https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721165#comment-16721165
]
Hudson commented on HBASE-20672:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> New metrics ReadRequestRate and WriteRequestRate
>
>
> Key: HBASE-20672
> URL: https://issues.apache.org/jira/browse/HBASE-20672
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Ankit Jain
>Assignee: Ankit Jain
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.10
>
> Attachments: HBASE-20672.branch-1.001.patch,
> HBASE-20672.branch-1.002.patch, HBASE-20672.branch-2.001.patch,
> HBASE-20672.master.001.patch, HBASE-20672.master.002.patch,
> HBASE-20672.master.003.patch, hits1vs2.4.40.400.png
>
>
> Hbase currently provides counter read/write requests (ReadRequestCount,
> WriteRequestCount). That said it is not easy to use counter that reset only
> after a restart of the service, we would like to expose 2 new metrics in
> HBase to provide ReadRequestRate and WriteRequestRate at region server level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20558) Backport HBASE-17854 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721170#comment-16721170
]
Hudson commented on HBASE-20558:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Backport HBASE-17854 to branch-1
>
>
> Key: HBASE-20558
> URL: https://issues.apache.org/jira/browse/HBASE-20558
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
>Affects Versions: 1.4.4, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20558.branch-1.001.patch, report.html
>
>
> As part of HBASE-20555, HBASE-17854 is the third patch that is needed for
> backporting HBASE-18083
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do
[
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721164#comment-16721164
]
Hudson commented on HBASE-20770:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
> Issue Type: Bug
> Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 1.5.0, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster,
> Master log is in a continuous spew of logging output fililng disks. It is
> stuck making no progress but hard to tell because it is logging minutiae
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
> exit...
> 2018-06-21 01:19:12,823 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
> exit...
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning
[jira] [Commented] (HBASE-20858) port HBASE-20695 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721175#comment-16721175
]
Hudson commented on HBASE-20858:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> port HBASE-20695 to branch-1
>
>
> Key: HBASE-20858
> URL: https://issues.apache.org/jira/browse/HBASE-20858
> Project: HBase
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Fix For: 1.5.0, 1.3.3, 1.4.10
>
> Attachments: HBASE-20858.branch-1.001.patch,
> HBASE-20858.branch-1.002.patch
>
>
> port HBASE-20695 to branch-1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21126) Add ability for HBase Canary to ignore a configurable number of ZooKeeper down nodes
[
https://issues.apache.org/jira/browse/HBASE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721183#comment-16721183
]
Hudson commented on HBASE-21126:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Add ability for HBase Canary to ignore a configurable number of ZooKeeper
> down nodes
>
>
> Key: HBASE-21126
> URL: https://issues.apache.org/jira/browse/HBASE-21126
> Project: HBase
> Issue Type: Improvement
> Components: canary, Zookeeper
>Affects Versions: 1.0.0, 3.0.0, 2.0.0
>Reporter: David Manning
>Assignee: David Manning
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21126.branch-1.001.patch,
> HBASE-21126.master.001.patch, HBASE-21126.master.002.patch,
> HBASE-21126.master.003.patch, zookeeperCanaryLocalTestValidation.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When running org.apache.hadoop.hbase.tool.Canary with args -zookeeper
> -treatFailureAsError, the Canary will try to get a znode from each ZooKeeper
> server in the ensemble. If any server is unavailable or unresponsive, the
> canary will exit with a failure code.
> If we use the Canary to gauge server health, and alert accordingly, this can
> be too strict. For example, in a 5-node ZooKeeper cluster, having one node
> down is safe and expected in rolling upgrades/patches.
> This is a request to allow the Canary to take another parameter
> {code:java}
> -permittedZookeeperFailures {code}
> If N=1, in the 5-node ZooKeeper ensemble example, then the Canary will still
> pass if 4 ZooKeeper nodes are reachable, but fail if 3 or fewer are reachable.
> (This is my first Jira posting... sorry if I messed anything up.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21212) Wrong flush time when update flush metric
[
https://issues.apache.org/jira/browse/HBASE-21212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721184#comment-16721184
]
Hudson commented on HBASE-21212:
Results for branch branch-1.3
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Wrong flush time when update flush metric
> -
>
> Key: HBASE-21212
> URL: https://issues.apache.org/jira/browse/HBASE-21212
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0, 2.0.2
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Fix For: 3.0.0, 1.3.3, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21212.branch-2.0.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
