[jira] [Updated] (HBASE-20237) Put back getClosestRowBefore and throw UnsupportedOperation instead... for asynchbase client
[ https://issues.apache.org/jira/browse/HBASE-20237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20237: -- Status: Patch Available (was: Open) .001 add test for getClosestOrBefore flag and throw exception if found. > Put back getClosestRowBefore and throw UnsupportedOperation instead... for > asynchbase client > > > Key: HBASE-20237 > URL: https://issues.apache.org/jira/browse/HBASE-20237 > Project: HBase > Issue Type: Bug > Components: compatibility, Operability >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20237.branch-2.001.patch > > > asychbase can work against all hbase versions. This fact has saved us a few > times; e.g. LINE were able to migrate from 0.94 to 1.2 going in part via > asynchbase and its ability to work against all servers. > hbase2 breaks this ability of asynchbase. There is nothing for asynchbase to > figure definitively that it is talking to an hbase2 server (See HBASE-20225). > Lets add back something it can leverage. HBASE-13954 did a nice job purging > getClosestRowBefore. Lets put back an RPC stuff that throws an exception if > it gets called. Thats enough for asynchbase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20237) Put back getClosestRowBefore and throw UnsupportedOperation instead... for asynchbase client
[ https://issues.apache.org/jira/browse/HBASE-20237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20237: -- Attachment: HBASE-20237.branch-2.001.patch > Put back getClosestRowBefore and throw UnsupportedOperation instead... for > asynchbase client > > > Key: HBASE-20237 > URL: https://issues.apache.org/jira/browse/HBASE-20237 > Project: HBase > Issue Type: Bug > Components: compatibility, Operability >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20237.branch-2.001.patch > > > asychbase can work against all hbase versions. This fact has saved us a few > times; e.g. LINE were able to migrate from 0.94 to 1.2 going in part via > asynchbase and its ability to work against all servers. > hbase2 breaks this ability of asynchbase. There is nothing for asynchbase to > figure definitively that it is talking to an hbase2 server (See HBASE-20225). > Lets add back something it can leverage. HBASE-13954 did a nice job purging > getClosestRowBefore. Lets put back an RPC stuff that throws an exception if > it gets called. Thats enough for asynchbase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20237) Put back getClosestRowBefore and throw UnsupportedOperation instead... for asynchbase client
[
https://issues.apache.org/jira/browse/HBASE-20237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407490#comment-16407490
]
stack commented on HBASE-20237:
---
Change is small. Just add back the getClosest flags to the Get protobuf and
then check for them being set at the front door; if they are throw
UnknownProtocolException. I put the patch in place and then tried connection to
hbase with an old OpenTSDB and got the below:
{code}
2018-03-20 22:40:09,231 INFO [main] TSDB: Flushing compaction queue
2018-03-20 22:40:09,233 INFO [main] HBaseClient: Channel [id: 0xd9575594,
/127.0.0.1:56289 => /127.0.0.1:56264] is disconnecting: [id: 0xd9575594,
/127.0.0.1:56289 => /127.0.0.1:56264] DISCONNECT
2018-03-20 22:40:09,233 INFO [main] HBaseClient: Lost connection with the
.META. region
2018-03-20 22:40:09,236 INFO [main] TSDB: Completed shutting down the TSDB
Exception in thread "main" java.lang.RuntimeException: Initialization failed
at net.opentsdb.tools.TSDMain.main(TSDMain.java:237)
Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the
Deferreds failed, first exception:
at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169)
at
com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142)
at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36)
at
com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
at com.stumbleupon.async.Deferred.access$300(Deferred.java:430)
at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1284)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1005)
at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712)
at org.hbase.async.RegionClient.decode(RegionClient.java:1536)
at org.hbase.async.RegionClient.decode(RegionClient.java:88)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:485)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1226)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)
at
org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at
org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3678)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
[jira] [Commented] (HBASE-20130) Document the ports used by RS are changed to 16201/16301 when the RS is bound to localhost
[ https://issues.apache.org/jira/browse/HBASE-20130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407488#comment-16407488 ] stack commented on HBASE-20130: --- [~uagashe] You think this related to what you are working on sir? > Document the ports used by RS are changed to 16201/16301 when the RS is bound > to localhost > -- > > Key: HBASE-20130 > URL: https://issues.apache.org/jira/browse/HBASE-20130 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407479#comment-16407479
]
Vladimir Rodionov commented on HBASE-19441:
---
Why is this illegal import, [~yuzhih...@gmail.com]?
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch, HBASE-19441-v4.patch, HBASE-19441-v5.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20234) Expose in-memory compaction metrics
[
https://issues.apache.org/jira/browse/HBASE-20234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407478#comment-16407478
]
Anoop Sam John commented on HBASE-20234:
It will be good to see info abt Compacting Memstore like the pipeline length,
the frequency of in memory flush/ compactions and also how these ops help in
reducing the memstore size. When in memory flush or compaction happens, we
expect the resulting heap size to be lesser.
> Expose in-memory compaction metrics
> ---
>
> Key: HBASE-20234
> URL: https://issues.apache.org/jira/browse/HBASE-20234
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
>
> Hard to glean insight from how well in-memory compaction is doing currently.
> It dumps stats into the logs but better if they were available to a
> dashboard. This issue is about exposing a couple of helpful counts. There are
> already by-region metrics. We can add a few for in-memory compaction (Help me
> out [~anastas]... what counts would be best to expose).
> Flush related metrics include
> {code}
> Namespace_default_table_tsdb-tree_region_cfbf23e7330a1a2bbde031f9583d3415_metric_flushesQueuedCount:
> {
> description: "Number flushes requested/queued for this region",
> value: 0
> {
> description: "The number of cells flushed to disk",
> value: 0
> },
> {
> description: "The total amount of data flushed to disk, in bytes",
> value: 0
> },
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407474#comment-16407474
]
Hudson commented on HBASE-20169:
Results for branch branch-2
[build #510 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/510/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/510//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/510//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/510//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch,
> HBASE-20169.branch-2.004.patch, HBASE-20169.branch-2.005.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19994) Create a new class for RPC throttling exception, make it retryable.
[
https://issues.apache.org/jira/browse/HBASE-19994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407463#comment-16407463
]
Hadoop QA commented on HBASE-19994:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
4s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
5s{color} | {color:red} hbase-server: The patch generated 1 new + 36 unchanged
- 1 fixed = 37 total (was 37) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
19s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 8s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} hbase-client generated 1 new + 2 unchanged - 0 fixed =
3 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
1s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}159m
21s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}209m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19994 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915410/HBASE-19994-master-v04.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ad86ccae6f9e 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / acbdb86bb4 |
| maven | version: Apache Maven 3.5.3
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407456#comment-16407456
]
Hadoop QA commented on HBASE-19852:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
41s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
57s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m
11s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m
27s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
14s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19852 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915348/HBASE-19852.master.006.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile findbugs hbaseanti checkstyle |
| uname | Linux 51203e14d2b2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 17ac2fe9c1 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
|
[jira] [Commented] (HBASE-19024) Configurable default durability for synchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407440#comment-16407440 ] Anoop Sam John commented on HBASE-19024: Raise an issue for the asyncwal [~apurtell] , [~harshaldjain] ? > Configurable default durability for synchronous WAL > --- > > Key: HBASE-19024 > URL: https://issues.apache.org/jira/browse/HBASE-19024 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Vikas Vishwakarma >Assignee: Harshal Jain >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-19024-master.v10.patch, > HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch, > HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch, > HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch, > branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch, > master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch > > > At present we do not have an option to hsync WAL edits to the disk for better > durability. In our local tests we see 10-15% latency impact of using hsync > instead of hflush which is not very high. > We should have a configurable option to hysnc WAL edits instead of just > sync/hflush which will call the corresponding API on the hadoop side. > Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling > FSDataOutputStream sync/hflush on the hadoop side. This can be modified to > let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can > keep the default value to sync as the current behavior and hsync can be > enabled based on explicit configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20229) ConnectionImplementation.locateRegions() doesn't work correctly when region replication is on
[ https://issues.apache.org/jira/browse/HBASE-20229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-20229: --- Fix Version/s: 2.0.0 > ConnectionImplementation.locateRegions() doesn't work correctly when region > replication is on > - > > Key: HBASE-20229 > URL: https://issues.apache.org/jira/browse/HBASE-20229 > Project: HBase > Issue Type: Bug >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20229.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20229) ConnectionImplementation.locateRegions() doesn't work correctly when region replication is on
[ https://issues.apache.org/jira/browse/HBASE-20229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407434#comment-16407434 ] Anoop Sam John commented on HBASE-20229: So the issue is duplicated entries in the List been returned. Every region will repeat 3 times (3 = replicas count).. Got it. Was wondering some other issue. Pls add proper description and change the title accordingly. Your patch make sense. In test we create table with one region but 3 been replicas count. The list returned should contain 3 locations.. Pls add more asserts abt the replica id of returned regions also. (That one is primary/default and others are 1 and 2 replica id).. good one. > ConnectionImplementation.locateRegions() doesn't work correctly when region > replication is on > - > > Key: HBASE-20229 > URL: https://issues.apache.org/jira/browse/HBASE-20229 > Project: HBase > Issue Type: Bug >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Attachments: HBASE-20229.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19024) Configurable default durability for synchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19024: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Configurable default durability for synchronous WAL > --- > > Key: HBASE-19024 > URL: https://issues.apache.org/jira/browse/HBASE-19024 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Vikas Vishwakarma >Assignee: Harshal Jain >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-19024-master.v10.patch, > HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch, > HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch, > HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch, > branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch, > master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch > > > At present we do not have an option to hsync WAL edits to the disk for better > durability. In our local tests we see 10-15% latency impact of using hsync > instead of hflush which is not very high. > We should have a configurable option to hysnc WAL edits instead of just > sync/hflush which will call the corresponding API on the hadoop side. > Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling > FSDataOutputStream sync/hflush on the hadoop side. This can be modified to > let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can > keep the default value to sync as the current behavior and hsync can be > enabled based on explicit configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407427#comment-16407427
]
Ted Yu commented on HBASE-20090:
Integrated to master branch.
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> Observed the following in region server log (running on hadoop3 cluster):
> {code}
> 2018-02-26 16:06:50,044 WARN
> [RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 135ms
> 2018-02-26 16:06:50,049 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry
> org.apache.hadoop.hbase.regionserver.
> MemStoreFlusher$WakeupFlushThread@2adfadd7
> java.lang.IllegalStateException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:441)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:174)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Here is the Precondition from MemStoreFlusher#flushOneForGlobalPressure() :
> {code}
> Preconditions.checkState(
> (regionToFlush != null && regionToFlushSize > 0) ||
> bestRegionReplicaSize > 0);
> {code}
> When the Preconditions check fails, IllegalStateException would be raised.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("NOT flushing
[jira] [Comment Edited] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407425#comment-16407425
]
stack edited comment on HBASE-20090 at 3/21/18 3:56 AM:
Now I get it. Thanks. Let me backport to branch-2.0... After it goes in
elsewhere.
was (Author: stack):
Now I get it. Thanks. Let me backport...
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> Observed the following in region server log (running on hadoop3 cluster):
> {code}
> 2018-02-26 16:06:50,044 WARN
> [RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 135ms
> 2018-02-26 16:06:50,049 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry
> org.apache.hadoop.hbase.regionserver.
> MemStoreFlusher$WakeupFlushThread@2adfadd7
> java.lang.IllegalStateException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:441)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:174)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Here is the Precondition from MemStoreFlusher#flushOneForGlobalPressure() :
> {code}
> Preconditions.checkState(
> (regionToFlush != null && regionToFlushSize > 0) ||
> bestRegionReplicaSize > 0);
> {code}
> When the Preconditions check fails, IllegalStateException would be raised.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing &&
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407425#comment-16407425
]
stack commented on HBASE-20090:
---
Now I get it. Thanks. Let me backport...
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> Observed the following in region server log (running on hadoop3 cluster):
> {code}
> 2018-02-26 16:06:50,044 WARN
> [RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 135ms
> 2018-02-26 16:06:50,049 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry
> org.apache.hadoop.hbase.regionserver.
> MemStoreFlusher$WakeupFlushThread@2adfadd7
> java.lang.IllegalStateException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:441)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:174)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Here is the Precondition from MemStoreFlusher#flushOneForGlobalPressure() :
> {code}
> Preconditions.checkState(
> (regionToFlush != null && regionToFlushSize > 0) ||
> bestRegionReplicaSize > 0);
> {code}
> When the Preconditions check fails, IllegalStateException would be raised.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
>
[jira] [Commented] (HBASE-20204) Add locking to RefreshFileConnections in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407350#comment-16407350
]
Hadoop QA commented on HBASE-20204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
9s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
1s{color} | {color:red} hbase-server: The patch generated 2 new + 7 unchanged -
0 fixed = 9 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 45s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 28s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.io.hfile.bucket.TestFileIOEngine |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20204 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915400/HBASE-20204.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 37cc0fc76fa3 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / acbdb86bb4 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/12046/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/12046/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-20232) [LOGGING] Formatting around close and flush
[
https://issues.apache.org/jira/browse/HBASE-20232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407348#comment-16407348
]
Hudson commented on HBASE-20232:
Results for branch branch-2.0
[build #65 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/65/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/65//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/65//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/65//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [LOGGING] Formatting around close and flush
> ---
>
> Key: HBASE-20232
> URL: https://issues.apache.org/jira/browse/HBASE-20232
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20232.branch-2.001.patch
>
>
> More log formatting. Fix issue where we had a param too many identified in
> parent issue. Remove some redundancy logging around flush and close. Use
> encoded regionname instead of full beast. Use SLF4J params. Remove some if
> debug tests. Use same formatter everywhere in flush when printing out data
> sizes (we had two types). A miscellany.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[ https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407333#comment-16407333 ] Chia-Ping Tsai commented on HBASE-20212: Thank [~yuzhih...@gmail.com]. Try patch for branch-2 > Make all Public classes have InterfaceAudience category > --- > > Key: HBASE-20212 > URL: https://issues.apache.org/jira/browse/HBASE-20212 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.1.0 > > Attachments: HBASE-20212.branch-2.v0.patch, HBASE-20212.v0.patch > > > The tasks will be resolved are shown below. > # add warbucks-maven-plugin to root pom > # make sure all sub modules ref the warbucks-maven-plugin > # remove old checker (TestInterfaceAudienceAnnotations) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20212) Make all Public classes have InterfaceAudience category
[ https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-20212: --- Attachment: HBASE-20212.branch-2.v0.patch > Make all Public classes have InterfaceAudience category > --- > > Key: HBASE-20212 > URL: https://issues.apache.org/jira/browse/HBASE-20212 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.1.0 > > Attachments: HBASE-20212.branch-2.v0.patch, HBASE-20212.v0.patch > > > The tasks will be resolved are shown below. > # add warbucks-maven-plugin to root pom > # make sure all sub modules ref the warbucks-maven-plugin > # remove old checker (TestInterfaceAudienceAnnotations) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407306#comment-16407306
]
Ted Yu commented on HBASE-19441:
Among the checkstyle warnings, this must be addressed:
{code}
./hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupManager.java:44:import
com.google.common.util.concurrent.Uninterruptibles;:1: Import from illegal
package - com.google.common.util.concurrent.Uninterruptibles. [IllegalImport]
{code}
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch, HBASE-19441-v4.patch, HBASE-19441-v5.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407303#comment-16407303
]
Hadoop QA commented on HBASE-19441:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hbase-backup: The patch generated 5 new + 0 unchanged
- 0 fixed = 5 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
59s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m
15s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m
43s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
10s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19441 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915396/HBASE-19441-v5.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4c675205ed44 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / acbdb86bb4 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
[jira] [Commented] (HBASE-20232) [LOGGING] Formatting around close and flush
[
https://issues.apache.org/jira/browse/HBASE-20232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407300#comment-16407300
]
Hudson commented on HBASE-20232:
Results for branch branch-2
[build #509 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/509/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/509//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/509//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/509//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [LOGGING] Formatting around close and flush
> ---
>
> Key: HBASE-20232
> URL: https://issues.apache.org/jira/browse/HBASE-20232
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20232.branch-2.001.patch
>
>
> More log formatting. Fix issue where we had a param too many identified in
> parent issue. Remove some redundancy logging around flush and close. Use
> encoded regionname instead of full beast. Use SLF4J params. Remove some if
> debug tests. Use same formatter everywhere in flush when printing out data
> sizes (we had two types). A miscellany.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407291#comment-16407291
]
Hadoop QA commented on HBASE-19343:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
3s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
26s{color} | {color:red} hbase-server in branch-1 failed with JDK v1.8.0_163.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} hbase-server in branch-1 failed with JDK v1.7.0_171.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} branch-1 passed with JDK v1.8.0_163 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
14s{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_163.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s{color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_163. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
15s{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_171.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_171. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
38s{color} | {color:red} The patch causes 44 errors with Hadoop v2.4.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
36s{color} | {color:red} The patch causes 44 errors with Hadoop v2.5.2. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK v1.8.0_163 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}139m 5s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterBalanceThrottling |
| | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles |
| | hadoop.hbase.replication.regionserver.TestGlobalThrottler |
| | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint |
| |
[jira] [Commented] (HBASE-20229) ConnectionImplementation.locateRegions() doesn't work correctly when region replication is on
[
https://issues.apache.org/jira/browse/HBASE-20229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407290#comment-16407290
]
Toshihiro Suzuki commented on HBASE-20229:
--
[~anoop.hbase] [~huaxiang] Sorry for my lack of explanation.
The issue is when region replication is on,
ConnectionImplementation.locateRegions() returns wrong results.
In the test in my patch, the table should have 1 primary region + 2
secondaries, but ConnectionImplementation.locateRegions() returns 9 regions. I
think this is because the following code calls locateRegion() even for replica
regions and then the result tripled.
{code}
for (RegionInfo regionInfo : regions) {
RegionLocations list = locateRegion(tableName, regionInfo.getStartKey(),
useCache, true);
if (list != null) {
for (HRegionLocation loc : list.getRegionLocations()) {
if (loc != null) {
locations.add(loc);
}
}
}
{code}
The fix in my patch is to make it call locateRegion() only for a primary region.
Thanks
> ConnectionImplementation.locateRegions() doesn't work correctly when region
> replication is on
> -
>
> Key: HBASE-20229
> URL: https://issues.apache.org/jira/browse/HBASE-20229
> Project: HBase
> Issue Type: Bug
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-20229.master.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20237) Put back getClosestRowBefore and throw UnsupportedOperation instead... for asynchbase client
stack created HBASE-20237: - Summary: Put back getClosestRowBefore and throw UnsupportedOperation instead... for asynchbase client Key: HBASE-20237 URL: https://issues.apache.org/jira/browse/HBASE-20237 Project: HBase Issue Type: Bug Components: compatibility, Operability Reporter: stack Assignee: stack Fix For: 2.0.0 asychbase can work against all hbase versions. This fact has saved us a few times; e.g. LINE were able to migrate from 0.94 to 1.2 going in part via asynchbase and its ability to work against all servers. hbase2 breaks this ability of asynchbase. There is nothing for asynchbase to figure definitively that it is talking to an hbase2 server (See HBASE-20225). Lets add back something it can leverage. HBASE-13954 did a nice job purging getClosestRowBefore. Lets put back an RPC stuff that throws an exception if it gets called. Thats enough for asynchbase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19024) Configurable default durability for synchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19024: --- Environment: (was: ) Release Note: The default durability setting for the synchronous WAL is Durability.SYNC_WAL, which triggers HDFS hflush() to flush edits to the datanodes. We also support Durability.FSYNC_WAL, which instead triggers HDFS hsync() to flush _and_ fsync edits. This change introduces the new configuration setting "hbase.wal.hsync", defaulting to FALSE, that if set to TRUE changes the default durability setting for the synchronous WAL to FSYNC_WAL. > Configurable default durability for synchronous WAL > --- > > Key: HBASE-19024 > URL: https://issues.apache.org/jira/browse/HBASE-19024 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Vikas Vishwakarma >Assignee: Harshal Jain >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-19024-master.v10.patch, > HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch, > HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch, > HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch, > branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch, > master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch > > > At present we do not have an option to hsync WAL edits to the disk for better > durability. In our local tests we see 10-15% latency impact of using hsync > instead of hflush which is not very high. > We should have a configurable option to hysnc WAL edits instead of just > sync/hflush which will call the corresponding API on the hadoop side. > Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling > FSDataOutputStream sync/hflush on the hadoop side. This can be modified to > let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can > keep the default value to sync as the current behavior and hsync can be > enabled based on explicit configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19994) Create a new class for RPC throttling exception, make it retryable.
[
https://issues.apache.org/jira/browse/HBASE-19994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun updated HBASE-19994:
-
Attachment: HBASE-19994-master-v04.patch
> Create a new class for RPC throttling exception, make it retryable.
>
>
> Key: HBASE-19994
> URL: https://issues.apache.org/jira/browse/HBASE-19994
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Attachments: HBASE-19994-master-v01.patch,
> HBASE-19994-master-v02.patch, HBASE-19994-master-v03.patch,
> HBASE-19994-master-v04.patch
>
>
> Based on a discussion at dev mailing list.
>
> {code:java}
> Thanks Andrew.
> +1 for the second option, I will create a jira for this change.
> Huaxiang
> On Feb 9, 2018, at 1:09 PM, Andrew Purtell wrote:
> We have
> public class ThrottlingException extends QuotaExceededException
> public class QuotaExceededException extends DoNotRetryIOException
> Let the storage quota limits throw QuotaExceededException directly (based
> on DNRIOE). That seems fine.
> However, ThrottlingException is thrown as a result of a temporal quota,
> so it is inappropriate for this to inherit from DNRIOE, it should inherit
> IOException instead so the client is allowed to retry until successful, or
> until the retry policy is exhausted.
> We are in a bit of a pickle because we've released with this inheritance
> hierarchy, so to change it we will need a new minor, or we will want to
> deprecate ThrottlingException and use a new exception class instead, one
> which does not inherit from DNRIOE.
> On Feb 7, 2018, at 9:25 AM, Huaxiang Sun wrote:
> Hi Mike,
> You are right. For rpc throttling, definitely it is retryable. For storage
> quota, I think it will be fail faster (non-retryable).
> We probably need to separate these two types of exceptions, I will do some
> more research and follow up.
> Thanks,
> Huaxiang
> On Feb 7, 2018, at 9:16 AM, Mike Drob wrote:
> I think, philosophically, there can be two kinds of QEE -
> For throttling, we can retry. The quota is a temporal quota - you have done
> too many operations this minute, please try again next minute and
> everything will work.
> For storage, we shouldn't retry. The quota is a fixed quote - you have
> exceeded your allotted disk space, please do not try again until you have
> remedied the situation.
> Our current usage conflates the two, sometimes it is correct, sometimes not.
> On Wed, Feb 7, 2018 at 11:00 AM, Huaxiang Sun wrote:
> Hi Stack,
> I run into a case that a mapreduce job in hive cannot finish because
> it runs into a QEE.
> I need to look into the hive mr task to see if QEE is not handled
> correctly in hbase code or in hive code.
> I am thinking that if QEE is a retryable exception, then it should be
> taken care of by the hbase code.
> I will check more and report back.
> Thanks,
> Huaxiang
> On Feb 7, 2018, at 8:23 AM, Stack wrote:
> QEE being a DNRIOE seems right on the face of it.
> But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a
> throttling scenario Huaxiang?
> Thanks,
> S
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19994) Create a new class for RPC throttling exception, make it retryable.
[
https://issues.apache.org/jira/browse/HBASE-19994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407275#comment-16407275
]
huaxiang sun commented on HBASE-19994:
--
v4 patch addressing all warnings.
> Create a new class for RPC throttling exception, make it retryable.
>
>
> Key: HBASE-19994
> URL: https://issues.apache.org/jira/browse/HBASE-19994
> Project: HBase
> Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Attachments: HBASE-19994-master-v01.patch,
> HBASE-19994-master-v02.patch, HBASE-19994-master-v03.patch,
> HBASE-19994-master-v04.patch
>
>
> Based on a discussion at dev mailing list.
>
> {code:java}
> Thanks Andrew.
> +1 for the second option, I will create a jira for this change.
> Huaxiang
> On Feb 9, 2018, at 1:09 PM, Andrew Purtell wrote:
> We have
> public class ThrottlingException extends QuotaExceededException
> public class QuotaExceededException extends DoNotRetryIOException
> Let the storage quota limits throw QuotaExceededException directly (based
> on DNRIOE). That seems fine.
> However, ThrottlingException is thrown as a result of a temporal quota,
> so it is inappropriate for this to inherit from DNRIOE, it should inherit
> IOException instead so the client is allowed to retry until successful, or
> until the retry policy is exhausted.
> We are in a bit of a pickle because we've released with this inheritance
> hierarchy, so to change it we will need a new minor, or we will want to
> deprecate ThrottlingException and use a new exception class instead, one
> which does not inherit from DNRIOE.
> On Feb 7, 2018, at 9:25 AM, Huaxiang Sun wrote:
> Hi Mike,
> You are right. For rpc throttling, definitely it is retryable. For storage
> quota, I think it will be fail faster (non-retryable).
> We probably need to separate these two types of exceptions, I will do some
> more research and follow up.
> Thanks,
> Huaxiang
> On Feb 7, 2018, at 9:16 AM, Mike Drob wrote:
> I think, philosophically, there can be two kinds of QEE -
> For throttling, we can retry. The quota is a temporal quota - you have done
> too many operations this minute, please try again next minute and
> everything will work.
> For storage, we shouldn't retry. The quota is a fixed quote - you have
> exceeded your allotted disk space, please do not try again until you have
> remedied the situation.
> Our current usage conflates the two, sometimes it is correct, sometimes not.
> On Wed, Feb 7, 2018 at 11:00 AM, Huaxiang Sun wrote:
> Hi Stack,
> I run into a case that a mapreduce job in hive cannot finish because
> it runs into a QEE.
> I need to look into the hive mr task to see if QEE is not handled
> correctly in hbase code or in hive code.
> I am thinking that if QEE is a retryable exception, then it should be
> taken care of by the hbase code.
> I will check more and report back.
> Thanks,
> Huaxiang
> On Feb 7, 2018, at 8:23 AM, Stack wrote:
> QEE being a DNRIOE seems right on the face of it.
> But if throttling, a DNRIOE is inappropriate. Where you seeing a QEE in a
> throttling scenario Huaxiang?
> Thanks,
> S
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20236) [locking] Write-time worst offenders
stack created HBASE-20236:
-
Summary: [locking] Write-time worst offenders
Key: HBASE-20236
URL: https://issues.apache.org/jira/browse/HBASE-20236
Project: HBase
Issue Type: Sub-task
Components: Performance
Affects Versions: 2.0.0-beta-2
Reporter: stack
Messing w/ my new toy, here are worst offenders locking; they must be bad if
they show up in this sampling profiler:
{code}
7 Total: 769321884622 (99.24%) samples: 2965
8 [ 0] java.util.concurrent.Semaphore$NonfairSync
9 [ 1] sun.misc.Unsafe.park
10 [ 2] java.util.concurrent.locks.LockSupport.park
11 [ 3]
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
12 [ 4]
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly
13 [ 5]
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
14 [ 6] java.util.concurrent.Semaphore.acquire
15 [ 7]
org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner
16 [ 8] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
17
18 Total: 4284274263 (0.55%) samples: 23543
19 [ 0] org.apache.hadoop.hbase.regionserver.MutableSegment
20 [ 1] org.apache.hadoop.hbase.ByteBufferKeyValue.getSequenceId
21 [ 2] org.apache.hadoop.hbase.regionserver.Segment.updateMetaInfo
22 [ 3] org.apache.hadoop.hbase.regionserver.Segment.internalAdd
23 [ 4] org.apache.hadoop.hbase.regionserver.MutableSegment.add
24 [ 5] org.apache.hadoop.hbase.regionserver.AbstractMemStore.internalAdd
25 [ 6] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
26 [ 7] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
27 [ 8] org.apache.hadoop.hbase.regionserver.HStore.add
28 [ 9] org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore
29 [10] org.apache.hadoop.hbase.regionserver.HRegion.access$600
30 [11]
org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore
31 [12]
org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0
32 [13]
org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$442.1445825895.visit
33 [14]
org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations
34 [15]
org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore
35 [16]
org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore
36 [17] org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate
37 [18] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
38 [19] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
39 [20] org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp
40 [21] org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp
41 [22]
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation
42 [23] org.apache.hadoop.hbase.regionserver.RSRpcServices.multi
43 [24]
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod
44 [25] org.apache.hadoop.hbase.ipc.RpcServer.call
45 [26] org.apache.hadoop.hbase.ipc.CallRunner.run
46 [27] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
47 [28] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
48
49 Total: 717708856 (0.09%) samples: 214
50 [ 0] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
51 [ 1] sun.misc.Unsafe.park
52 [ 2] java.util.concurrent.locks.LockSupport.park
53 [ 3]
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
54 [ 4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
55 [ 5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
56 [ 6] java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock
57 [ 7] org.apache.hadoop.hbase.regionserver.HRegion.blockUpdates
58 [ 8]
org.apache.hadoop.hbase.regionserver.RegionServicesForStores.blockUpdates
59 [ 9] org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory
60 [10]
org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run
61 [11] java.util.concurrent.ThreadPoolExecutor.runWorker
62 [12] java.util.concurrent.ThreadPoolExecutor$Worker.run
63 [13] java.lang.Thread.run
...
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19024) Configurable default durability for synchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19024: --- Summary: Configurable default durability for synchronous WAL (was: provide a configurable option to hsync WAL edits to the disk for better durability) > Configurable default durability for synchronous WAL > --- > > Key: HBASE-19024 > URL: https://issues.apache.org/jira/browse/HBASE-19024 > Project: HBase > Issue Type: Improvement > Components: wal > Environment: >Reporter: Vikas Vishwakarma >Assignee: Harshal Jain >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-19024-master.v10.patch, > HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch, > HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch, > HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch, > branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch, > master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch > > > At present we do not have an option to hsync WAL edits to the disk for better > durability. In our local tests we see 10-15% latency impact of using hsync > instead of hflush which is not very high. > We should have a configurable option to hysnc WAL edits instead of just > sync/hflush which will call the corresponding API on the hadoop side. > Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling > FSDataOutputStream sync/hflush on the hadoop side. This can be modified to > let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can > keep the default value to sync as the current behavior and hsync can be > enabled based on explicit configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19024) provide a configurable option to hsync WAL edits to the disk for better durability
[ https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407271#comment-16407271 ] Andrew Purtell commented on HBASE-19024: Thanks [~anoop.hbase], will proceed > provide a configurable option to hsync WAL edits to the disk for better > durability > -- > > Key: HBASE-19024 > URL: https://issues.apache.org/jira/browse/HBASE-19024 > Project: HBase > Issue Type: Improvement > Components: wal > Environment: >Reporter: Vikas Vishwakarma >Assignee: Harshal Jain >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-19024-master.v10.patch, > HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch, > HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch, > HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch, > branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch, > master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch > > > At present we do not have an option to hsync WAL edits to the disk for better > durability. In our local tests we see 10-15% latency impact of using hsync > instead of hflush which is not very high. > We should have a configurable option to hysnc WAL edits instead of just > sync/hflush which will call the corresponding API on the hadoop side. > Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling > FSDataOutputStream sync/hflush on the hadoop side. This can be modified to > let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can > keep the default value to sync as the current behavior and hsync can be > enabled based on explicit configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20235) Follow up to commit HBASE-19364 for branch-2 and master.
[ https://issues.apache.org/jira/browse/HBASE-20235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun reassigned HBASE-20235: Assignee: huaxiang sun > Follow up to commit HBASE-19364 for branch-2 and master. > > > Key: HBASE-20235 > URL: https://issues.apache.org/jira/browse/HBASE-20235 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 3.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > > HBASE-19364 needs to be committed to branch-2 and master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20235) Follow up to commit HBASE-19364 for branch-2 and master.
huaxiang sun created HBASE-20235: Summary: Follow up to commit HBASE-19364 for branch-2 and master. Key: HBASE-20235 URL: https://issues.apache.org/jira/browse/HBASE-20235 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 3.0.0 Reporter: huaxiang sun HBASE-19364 needs to be committed to branch-2 and master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407254#comment-16407254
]
Ted Yu commented on HBASE-20090:
bq. but we are still above the high-water mark?
At the moment of trigger of Precondition, the lower mark was crossed. See log:
{code}
2018-03-02 17:28:30,301 INFO
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
{code}
bq. Do you know why that is?
The write load from PE produced the above condition which lasted for some
period of time.
bq. There were two regions in this server that was under pressue?
No. Among the two regions I mentioned, one was under pressure.
0453f29030757eedb6e6a1c57e88c085 for TestTable was receiving writes and under
pressure (and splitting).
The region for atlas_janus table was dormant during the period when PE ran. It
had a zero-sized memstore.
Since the existing Precondition didn't account for this combination, the
assertion was triggered.
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> Observed the following in region server log (running on hadoop3 cluster):
> {code}
> 2018-02-26 16:06:50,044 WARN
> [RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 135ms
> 2018-02-26 16:06:50,049 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry
> org.apache.hadoop.hbase.regionserver.
> MemStoreFlusher$WakeupFlushThread@2adfadd7
> java.lang.IllegalStateException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:441)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:174)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Here is the Precondition from MemStoreFlusher#flushOneForGlobalPressure() :
> {code}
> Preconditions.checkState(
> (regionToFlush != null && regionToFlushSize > 0) ||
> bestRegionReplicaSize > 0);
> {code}
> When the Preconditions check fails, IllegalStateException would be raised.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up
[jira] [Commented] (HBASE-19364) Truncate_preserve fails with table when replica region > 1
[ https://issues.apache.org/jira/browse/HBASE-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407255#comment-16407255 ] huaxiang sun commented on HBASE-19364: -- Thanks [~apurtell]. I am going to create a new issue for branch-2 and master. > Truncate_preserve fails with table when replica region > 1 > -- > > Key: HBASE-19364 > URL: https://issues.apache.org/jira/browse/HBASE-19364 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Major > Fix For: 1.5.0, 1.4.3 > > Attachments: HBASE-19364-branch-1-v2.patch, > HBASE-19364-branch-1-v3.patch, HBASE-19364-branch-1.patch, > HBASE-19364-branch-1.patch > > > Root cause is same as HBASE-17319, here we need to exclude secondary regions > while reading meta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20229) ConnectionImplementation.locateRegions() doesn't work correctly when region replication is on
[ https://issues.apache.org/jira/browse/HBASE-20229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407252#comment-16407252 ] huaxiang sun commented on HBASE-20229: -- Please describe the issue, it seems that from the test that without the fix, the locations array size is 9 instead of the 3. > ConnectionImplementation.locateRegions() doesn't work correctly when region > replication is on > - > > Key: HBASE-20229 > URL: https://issues.apache.org/jira/browse/HBASE-20229 > Project: HBase > Issue Type: Bug >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Attachments: HBASE-20229.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20204) Add locking to RefreshFileConnections in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20204: -- Status: Patch Available (was: Open) > Add locking to RefreshFileConnections in BucketCache > > > Key: HBASE-20204 > URL: https://issues.apache.org/jira/browse/HBASE-20204 > Project: HBase > Issue Type: Bug > Components: BucketCache >Reporter: Zach York >Assignee: Zach York >Priority: Major > Attachments: HBASE-20204.master.001.patch > > > This is a follow-up to HBASE-20141 where [~anoop.hbase] suggested adding > locking for refreshing channels. > I have also seen this become an issue when a RS has to abort and it locks on > trying to flush out the remaining data to the cache (since cache on write was > turned on). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20204) Add locking to RefreshFileConnections in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20204: -- Affects Version/s: 1.4.3 2.0.0 > Add locking to RefreshFileConnections in BucketCache > > > Key: HBASE-20204 > URL: https://issues.apache.org/jira/browse/HBASE-20204 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.4.3 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Attachments: HBASE-20204.master.001.patch > > > This is a follow-up to HBASE-20141 where [~anoop.hbase] suggested adding > locking for refreshing channels. > I have also seen this become an issue when a RS has to abort and it locks on > trying to flush out the remaining data to the cache (since cache on write was > turned on). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20204) Add locking to RefreshFileConnections in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20204: -- Attachment: HBASE-20204.master.001.patch > Add locking to RefreshFileConnections in BucketCache > > > Key: HBASE-20204 > URL: https://issues.apache.org/jira/browse/HBASE-20204 > Project: HBase > Issue Type: Bug > Components: BucketCache >Reporter: Zach York >Assignee: Zach York >Priority: Major > Attachments: HBASE-20204.master.001.patch > > > This is a follow-up to HBASE-20141 where [~anoop.hbase] suggested adding > locking for refreshing channels. > I have also seen this become an issue when a RS has to abort and it locks on > trying to flush out the remaining data to the cache (since cache on write was > turned on). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407204#comment-16407204
]
stack commented on HBASE-20090:
---
Yeah, sorry, RN makes no sense to me still but the description update helped a
bunch. Thanks.
So, we avoid the precondition illegal state exception (including this in
description helped most) but we are still above the high-water mark? Do you
know why that is? There were two regions in this server that was under pressue?
One that had a zero-sized memstore and another that was currently splitting?
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> Observed the following in region server log (running on hadoop3 cluster):
> {code}
> 2018-02-26 16:06:50,044 WARN
> [RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 135ms
> 2018-02-26 16:06:50,049 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry
> org.apache.hadoop.hbase.regionserver.
> MemStoreFlusher$WakeupFlushThread@2adfadd7
> java.lang.IllegalStateException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:441)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:174)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Here is the Precondition from MemStoreFlusher#flushOneForGlobalPressure() :
> {code}
> Preconditions.checkState(
> (regionToFlush != null && regionToFlushSize > 0) ||
> bestRegionReplicaSize > 0);
> {code}
> When the Preconditions check fails, IllegalStateException would be raised.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407194#comment-16407194
]
Vladimir Rodionov commented on HBASE-19441:
---
Patch v5. Fixed indentation issues.
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch, HBASE-19441-v4.patch, HBASE-19441-v5.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-19441:
--
Attachment: HBASE-19441-v5.patch
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch, HBASE-19441-v4.patch, HBASE-19441-v5.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407182#comment-16407182
]
Ted Yu commented on HBASE-20090:
I have updated both description and Release Note.
Thanks
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> Observed the following in region server log (running on hadoop3 cluster):
> {code}
> 2018-02-26 16:06:50,044 WARN
> [RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 135ms
> 2018-02-26 16:06:50,049 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry
> org.apache.hadoop.hbase.regionserver.
> MemStoreFlusher$WakeupFlushThread@2adfadd7
> java.lang.IllegalStateException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:441)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:174)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Here is the Precondition from MemStoreFlusher#flushOneForGlobalPressure() :
> {code}
> Preconditions.checkState(
> (regionToFlush != null && regionToFlushSize > 0) ||
> bestRegionReplicaSize > 0);
> {code}
> When the Preconditions check fails, IllegalStateException would be raised.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
>
[jira] [Updated] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20090:
---
Release Note:
When there is concurrent region split, MemStoreFlusher may not find flushable
region if the only candidate region left hasn't received writes (resulting in 0
data size).
After this JIRA, such scenario wouldn't trigger Precondition assertion
(replaced by an if statement to see whether there is any flushable region).
If there is no flushable region, a DEBUG log would appear in region server log,
saying "Above memory mark but there is no flushable region".
was:
When there is concurrent region split, MemStoreFlusher may not find flushable
region if the only candidate region left hasn't received writes.
After this JIRA, such scenario wouldn't trigger Precondition assertion.
Instead, a DEBUG log would appear in region server log.
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> Observed the following in region server log (running on hadoop3 cluster):
> {code}
> 2018-02-26 16:06:50,044 WARN
> [RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16020]
> regionserver.MemStoreFlusher: Memstore is above high water mark and block
> 135ms
> 2018-02-26 16:06:50,049 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry
> org.apache.hadoop.hbase.regionserver.
> MemStoreFlusher$WakeupFlushThread@2adfadd7
> java.lang.IllegalStateException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:441)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:174)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Here is the Precondition from MemStoreFlusher#flushOneForGlobalPressure() :
> {code}
> Preconditions.checkState(
> (regionToFlush != null && regionToFlushSize > 0) ||
> bestRegionReplicaSize > 0);
> {code}
> When the Preconditions check fails, IllegalStateException would be raised.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
>
[jira] [Updated] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20090:
---
Description:
Copied the following from a comment since this was better description of the
race condition.
The original description was merged to the beginning of my first comment below.
Observed the following in region server log (running on hadoop3 cluster):
{code}
2018-02-26 16:06:50,044 WARN
[RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16020]
regionserver.MemStoreFlusher: Memstore is above high water mark and block 135ms
2018-02-26 16:06:50,049 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher:
Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.
MemStoreFlusher$WakeupFlushThread@2adfadd7
java.lang.IllegalStateException
at
org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:441)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:174)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:69)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
at java.lang.Thread.run(Thread.java:748)
{code}
Here is the Precondition from MemStoreFlusher#flushOneForGlobalPressure() :
{code}
Preconditions.checkState(
(regionToFlush != null && regionToFlushSize > 0) ||
bestRegionReplicaSize > 0);
{code}
When the Preconditions check fails, IllegalStateException would be raised.
With more debug logging, we can see the scenario where the exception was
triggered.
{code}
2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
compaction_queue=(0:0), split_queue=1
2018-03-02 17:28:30,098 DEBUG
[RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because info
size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
2018-03-02 17:28:30,296 INFO
[RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1] regionserver.MemStoreFlusher:
Flush thread woke up because memory above low water=381.5 M
2018-03-02 17:28:30,297 INFO
[RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1] regionserver.MemStoreFlusher:
region TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size
400432696
2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1] regionserver.MemStoreFlusher:
region atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher:
Flush of region atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
due to global heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal
Memstore Heap size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher:
wake up by WAKEUPFLUSH_INSTANCE
2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher:
Nothing to flush for
atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher:
Excluding unflushable region
atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
find a different region to flush.
{code}
Region 0453f29030757eedb6e6a1c57e88c085 was being split.
In HRegion#flushcache, the log from else branch can be seen in
20090-server-61260-01-07.log :
{code}
synchronized (writestate) {
if (!writestate.flushing && writestate.writesEnabled) {
this.writestate.flushing = true;
} else {
if (LOG.isDebugEnabled()) {
LOG.debug("NOT flushing memstore for region " + this
+ ", flushing=" + writestate.flushing + ", writesEnabled="
+ writestate.writesEnabled);
}
{code}
Meaning, region 0453f29030757eedb6e6a1c57e88c085 couldn't flush, leaving memory
pressure at high level.
When MemStoreFlusher ran to the following call, the region was no longer a
flush candidate:
{code}
HRegion bestFlushableRegion =
getBiggestMemStoreRegion(regionsBySize, excludedRegions, true);
{code}
So the other region,
atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. , was examined
next. Since the region was not receiving write, the (current) Precondition
check failed.
The proposed fix is to convert the Precondition to normal return.
was:
Copied the following from a
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407160#comment-16407160
]
Hadoop QA commented on HBASE-19441:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hbase-backup: The patch generated 10 new + 0 unchanged
- 0 fixed = 10 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
25s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 5s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
38s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19441 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915383/HBASE-19441-v4.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8f26608153e2 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / acbdb86bb4 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/12043/artifact/patchprocess/diff-checkstyle-hbase-backup.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12043/testReport/ |
| Max. process+thread count | 4620 (vs. ulimit of 1) |
| modules | C: hbase-backup U: hbase-backup |
| Console output |
[jira] [Updated] (HBASE-20203) [AMv2] CODE-BUG: Uncaught runtime exception for pid=...., state=SUCCESS; AssignProcedure
[
https://issues.apache.org/jira/browse/HBASE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20203:
--
Fix Version/s: 2.0.0
> [AMv2] CODE-BUG: Uncaught runtime exception for pid=, state=SUCCESS;
> AssignProcedure
>
>
> Key: HBASE-20203
> URL: https://issues.apache.org/jira/browse/HBASE-20203
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
>
> This is an odd one. Causes ITBLL to fail because region is offline.
> Two seconds after reporting Finished, successful assign, another thread tries
> to finish the Procedure. The second run messes us up.
> {code}
> 2018-03-14 11:04:07,987 INFO [PEWorker-1] procedure2.ProcedureExecutor:
> Finished pid=3600, ppid=3591, state=SUCCESS; AssignProcedure
> table=IntegrationTestBigLinkedList, region=b58e6e7c3b2e449f80533ea999707319
> in 4.4100sec
>
> 2018-03-14 11:04:10,600 INFO [PEWorker-2]
> procedure.MasterProcedureScheduler: pid=3600, ppid=3591, state=SUCCESS;
> AssignProcedure table=IntegrationTestBigLinkedList,
> region=b58e6e7c3b2e449f80533ea999707319,
> IntegrationTestBigLinkedList,\x9Ey\xE7\x9Ey\xE7\x9Ep,1521050540660.b58e6e7c3b2e449f80533ea999707319.
> 2018-03-14 11:04:10,606 ERROR [PEWorker-2] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=3600, ppid=3591, state=SUCCESS;
> AssignProcedure table=IntegrationTestBigLinkedList,
> region=b58e6e7c3b2e449f80533ea999707319
>
> java.lang.UnsupportedOperationException: Unhandled state
> REGION_TRANSITION_FINISH; there is no rollback for assignment unless we
> cancel the operation by dropping/disabling the table
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:345)
>
>
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:86)
>
>
>at
> org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:859)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1353)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1309)
>
>
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1178)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
>
>
>
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20203) [AMv2] CODE-BUG: Uncaught runtime exception for pid=...., state=SUCCESS; AssignProcedure
[
https://issues.apache.org/jira/browse/HBASE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407150#comment-16407150
]
stack commented on HBASE-20203:
---
Haven't seen this in recent runs. Keeping an eye on it.
> [AMv2] CODE-BUG: Uncaught runtime exception for pid=, state=SUCCESS;
> AssignProcedure
>
>
> Key: HBASE-20203
> URL: https://issues.apache.org/jira/browse/HBASE-20203
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
>
> This is an odd one. Causes ITBLL to fail because region is offline.
> Two seconds after reporting Finished, successful assign, another thread tries
> to finish the Procedure. The second run messes us up.
> {code}
> 2018-03-14 11:04:07,987 INFO [PEWorker-1] procedure2.ProcedureExecutor:
> Finished pid=3600, ppid=3591, state=SUCCESS; AssignProcedure
> table=IntegrationTestBigLinkedList, region=b58e6e7c3b2e449f80533ea999707319
> in 4.4100sec
>
> 2018-03-14 11:04:10,600 INFO [PEWorker-2]
> procedure.MasterProcedureScheduler: pid=3600, ppid=3591, state=SUCCESS;
> AssignProcedure table=IntegrationTestBigLinkedList,
> region=b58e6e7c3b2e449f80533ea999707319,
> IntegrationTestBigLinkedList,\x9Ey\xE7\x9Ey\xE7\x9Ep,1521050540660.b58e6e7c3b2e449f80533ea999707319.
> 2018-03-14 11:04:10,606 ERROR [PEWorker-2] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=3600, ppid=3591, state=SUCCESS;
> AssignProcedure table=IntegrationTestBigLinkedList,
> region=b58e6e7c3b2e449f80533ea999707319
>
> java.lang.UnsupportedOperationException: Unhandled state
> REGION_TRANSITION_FINISH; there is no rollback for assignment unless we
> cancel the operation by dropping/disabling the table
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:345)
>
>
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.rollback(RegionTransitionProcedure.java:86)
>
>
>at
> org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:859)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1353)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1309)
>
>
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1178)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
>
>
>
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20090) Properly handle Preconditions check failure in MemStoreFlusher$FlushHandler.run
[
https://issues.apache.org/jira/browse/HBASE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407149#comment-16407149
]
stack commented on HBASE-20090:
---
So, to understand the release note I have to read the description ... which I
have and I still don't get it -- I reread it and it talks about exceptions of
which there are none in the description and it talks of preconditions but I
don't see it the code snippet provided. I look at the patch and it adds a new
check. What are we avoiding with this patch? Please put it in the RN. I'm
trying to avoid having to read this whole issue to figure whether I need this
patch in 2.0 or not. Thanks.
> Properly handle Preconditions check failure in
> MemStoreFlusher$FlushHandler.run
> ---
>
> Key: HBASE-20090
> URL: https://issues.apache.org/jira/browse/HBASE-20090
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20090-server-61260-01-07.log, 20090.v10.txt,
> 20090.v10.txt, 20090.v6.txt, 20090.v7.txt, 20090.v8.txt, 20090.v9.txt
>
>
> Copied the following from a comment since this was better description of the
> race condition.
> The original description was merged to the beginning of my first comment
> below.
> With more debug logging, we can see the scenario where the exception was
> triggered.
> {code}
> 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit:
> Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.,
> compaction_queue=(0:0), split_queue=1
> 2018-03-02 17:28:30,098 DEBUG
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
> info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1
> 2018-03-02 17:28:30,296 INFO
> [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush thread woke up because memory above low
> water=381.5 M
> 2018-03-02 17:28:30,297 INFO
> [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020]
> regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696
> 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Flush of region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global
> heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap
> size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Nothing to flush for
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae.
> 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Excluding unflushable region
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. -trying to
> find a different region to flush.
> {code}
> Region 0453f29030757eedb6e6a1c57e88c085 was being split.
> In HRegion#flushcache, the log from else branch can be seen in
> 20090-server-61260-01-07.log :
> {code}
> synchronized (writestate) {
> if (!writestate.flushing && writestate.writesEnabled) {
> this.writestate.flushing = true;
> } else {
> if (LOG.isDebugEnabled()) {
> LOG.debug("NOT flushing memstore for region " + this
> + ", flushing=" + writestate.flushing + ", writesEnabled="
> + writestate.writesEnabled);
> }
> {code}
> Meaning, region 0453f29030757eedb6e6a1c57e88c085 couldn't flush, leaving
> memory pressure at high level.
> When MemStoreFlusher ran to the following call, the region was no longer a
> flush candidate:
> {code}
> HRegion bestFlushableRegion =
> getBiggestMemStoreRegion(regionsBySize, excludedRegions, true);
> {code}
> So the other region,
> atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. , was examined
> next. Since the region was not receiving write, the (current) Precondition
> check failed.
> The proposed fix is to convert the Precondition to normal return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20234) Expose in-memory compaction metrics
stack created HBASE-20234:
-
Summary: Expose in-memory compaction metrics
Key: HBASE-20234
URL: https://issues.apache.org/jira/browse/HBASE-20234
Project: HBase
Issue Type: Sub-task
Reporter: stack
Hard to glean insight from how well in-memory compaction is doing currently. It
dumps stats into the logs but better if they were available to a dashboard.
This issue is about exposing a couple of helpful counts. There are already
by-region metrics. We can add a few for in-memory compaction (Help me out
[~anastas]... what counts would be best to expose).
Flush related metrics include
{code}
Namespace_default_table_tsdb-tree_region_cfbf23e7330a1a2bbde031f9583d3415_metric_flushesQueuedCount:
{
description: "Number flushes requested/queued for this region",
value: 0
{
description: "The number of cells flushed to disk",
value: 0
},
{
description: "The total amount of data flushed to disk, in bytes",
value: 0
},
...
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406572#comment-16406572 ] stack edited comment on HBASE-20188 at 3/20/18 10:12 PM: - [~anastas] yeah, lets expose metrics so we can 'watch' the in-memory compactions across time. I opened HBASE-20234. was (Author: stack): [~anastas] yeah, lets expose metrics so we can 'watch' the in-memory compactions across time. File an issue? > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-19639.
---
Resolution: Fixed
Resolving as fixed now. Was able to go to 25B a day or so ago. The metric
hbase.regionserver.ipc.exceptions.RegionTooBusyException is staying mostly zero
over an ITBLL run.
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10033ms, replay=524ops
> 2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient]
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities
> for 6 ms, close active connection. Will reconnect next time when there
> are new requests.
> 2017-12-26 19:24:39,122 WARN
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407107#comment-16407107
]
stack commented on HBASE-19639:
---
I fixed the logging formatting in a sub-issue.
G1GC as default is another project as is changing MSLAB default.
I had the memstore multiplier at 8. Default is 4. I set it back to the default
for the last few runs.
I found this metric:
{{hbase.regionserver.ipc.exceptions.RegionTooBusyException}}. Its a count of
how many {{RegionTooBusyException}} are thrown on the server. It says none for
the last few runs. It looks like this metric is hooked up properly since other
exception counts come and go and I just noticed a non-zero on a new run. So, I
think that I'm over this issue now. Return to defaults and fix ups in
assignment done over last month or so allow ITBLL to make progress not stalled
on RegionTooBusyException.
I'll resolve this issue as no longer valid. Lets carry on the tuning, etc.,
discussion over in the perf and scale issues, HBASE-20179 and HBASE-20188
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
>
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407088#comment-16407088
]
Vladimir Rodionov commented on HBASE-19441:
---
Patch v4.
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch, HBASE-19441-v4.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-19441:
--
Attachment: HBASE-19441-v4.patch
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch, HBASE-19441-v4.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406961#comment-16406961
]
Ted Yu commented on HBASE-19441:
{code}
+public class ExclusiveOperationException extends IOException {
{code}
Please add audience annotation (Private).
Otherwise looks good.
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406948#comment-16406948
]
stack commented on HBASE-20169:
---
Pushed to branch-2.0, branch-2, and master. Leaving open to see if this helps
w/ flakey tests.
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch,
> HBASE-20169.branch-2.004.patch, HBASE-20169.branch-2.005.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20231) Not able to delete column family from a row using RemoteHTable
[
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406940#comment-16406940
]
Umesh Agashe commented on HBASE-20231:
--
+1
> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-20231-branch-1.patch, HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:"
> instead of "/t1/r2/cf1".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20233) [metrics] Ill-formatted numRegions metric in "Hadoop:service=HBase,name=RegionServer,sub=Regions" mbean
stack created HBASE-20233:
-
Summary: [metrics] Ill-formatted numRegions metric in
"Hadoop:service=HBase,name=RegionServer,sub=Regions" mbean
Key: HBASE-20233
URL: https://issues.apache.org/jira/browse/HBASE-20233
Project: HBase
Issue Type: Bug
Components: metrics
Environment: Noticed this poking at metrics. The MBean
"Hadoop:service=HBase,name=RegionServer,sub=Regions" carries per region
metrics. They are all formatted like so:
{code}
Namespace_default_table_IntegrationTestBigLinkedList_metric_incrementTime_98th_percentile:
0,
Namespace_default_table_IntegrationTestBigLinkedList_metric_incrementTime_99th_percentile:
0,
Namespace_default_table_IntegrationTestBigLinkedList_metric_incrementTime_99.9th_percentile:
0,
Namespace_default_table_IntegrationTestBigLinkedList_metric_appendTime_num_ops:
0,
Namespace_default_table_IntegrationTestBigLinkedList_metric_appendTime_min: 0,
Namespace_default_table_IntegrationTestBigLinkedList_metric_appendTime_max: 0,
Namespace_default_table_IntegrationTestBigLinkedList_metric_appendTime_mean: 0,
{code}
In the middle of them all is a metric named...
{code}
numRegions: 15,
{code}
It has a different format and it is out of scope when compared to the other
metrics in this mbean; it belongs better elsewhere, perhaps in the Server bean.
I noticed it because it breaks the parse done by tcollector scraping our
metrics from /jmx
It was added long ago in HBASE-14166
Reporter: stack
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406897#comment-16406897
]
Hadoop QA commented on HBASE-20169:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 9m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
58s{color} | {color:red} hbase-server: The patch generated 3 new + 6 unchanged
- 0 fixed = 9 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
27s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
31s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}171m
19s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}233m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-20169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915342/HBASE-20169.branch-2.005.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
[jira] [Commented] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406878#comment-16406878
]
Pankaj Kumar commented on HBASE-19343:
--
Attached updated V2 patch based on the latest code.
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-19343-branch-1-v2.patch,
> HBASE-19343-branch-1.patch, Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
>
[jira] [Updated] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-19343:
-
Attachment: HBASE-19343-branch-1-v2.patch
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-19343-branch-1-v2.patch,
> HBASE-19343-branch-1.patch, Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
>
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406867#comment-16406867
]
Vladimir Rodionov commented on HBASE-19441:
---
Patch v3.
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-19441:
--
Attachment: HBASE-19441-v3.patch
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch,
> HBASE-19441-v3.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406767#comment-16406767
]
Kevin Risden commented on HBASE-19852:
--
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-install-plugin:2.5.2:install (default-install)
on project hbase-thrift: Failed to install metadata
org.apache.hbase:hbase-thrift:3.0.0-SNAPSHOT/maven-metadata.xml: Could not
parse metadata
/home/jenkins/.m2/repository/org/apache/hbase/hbase-thrift/3.0.0-SNAPSHOT/maven-metadata-local.xml:
in epilog non whitespace content is not allowed but got / (position: END_TAG
seen ...\n/... @25:2) -> [Help 1]{noformat}
Any idea what this means?
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20229) ConnectionImplementation.locateRegions() doesn't work correctly when region replication is on
[ https://issues.apache.org/jira/browse/HBASE-20229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406752#comment-16406752 ] Anoop Sam John commented on HBASE-20229: What is the issue you are facing? Any chance for a test also for this? Ya it make sense to do locateRegion() call only on primary region. > ConnectionImplementation.locateRegions() doesn't work correctly when region > replication is on > - > > Key: HBASE-20229 > URL: https://issues.apache.org/jira/browse/HBASE-20229 > Project: HBase > Issue Type: Bug >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Attachments: HBASE-20229.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20232) [LOGGING] Formatting around close and flush
[ https://issues.apache.org/jira/browse/HBASE-20232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-20232. --- Resolution: Fixed Assignee: stack Fix Version/s: 2.0.0 What I pushed to branch-2.0, branch-2, and master. > [LOGGING] Formatting around close and flush > --- > > Key: HBASE-20232 > URL: https://issues.apache.org/jira/browse/HBASE-20232 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20232.branch-2.001.patch > > > More log formatting. Fix issue where we had a param too many identified in > parent issue. Remove some redundancy logging around flush and close. Use > encoded regionname instead of full beast. Use SLF4J params. Remove some if > debug tests. Use same formatter everywhere in flush when printing out data > sizes (we had two types). A miscellany. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20232) [LOGGING] Formatting around close and flush
[ https://issues.apache.org/jira/browse/HBASE-20232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20232: -- Attachment: HBASE-20232.branch-2.001.patch > [LOGGING] Formatting around close and flush > --- > > Key: HBASE-20232 > URL: https://issues.apache.org/jira/browse/HBASE-20232 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Priority: Major > Attachments: HBASE-20232.branch-2.001.patch > > > More log formatting. Fix issue where we had a param too many identified in > parent issue. Remove some redundancy logging around flush and close. Use > encoded regionname instead of full beast. Use SLF4J params. Remove some if > debug tests. Use same formatter everywhere in flush when printing out data > sizes (we had two types). A miscellany. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20232) [LOGGING] Formatting around close and flush
stack created HBASE-20232: - Summary: [LOGGING] Formatting around close and flush Key: HBASE-20232 URL: https://issues.apache.org/jira/browse/HBASE-20232 Project: HBase Issue Type: Sub-task Reporter: stack More log formatting. Fix issue where we had a param too many identified in parent issue. Remove some redundancy logging around flush and close. Use encoded regionname instead of full beast. Use SLF4J params. Remove some if debug tests. Use same formatter everywhere in flush when printing out data sizes (we had two types). A miscellany. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406740#comment-16406740
]
Anoop Sam John commented on HBASE-19639:
For us the default memstore global size is 40% and BC is 40%. When it is R+W
workload with this default configs, it is like the working size of the server
is always 80%+ of Xmx. The default InitialHeapOccupancyPercentage (IHOP) is
45% for G1GC. The larger value for this does not make G1GC meaningful IMHO.
G1GC is basically for predictable GC pause. With these setup, we will get
more GC pauses.. So tuning Xmx and this IHOP is very key for G1GC based usage.
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406727#comment-16406727
]
Hadoop QA commented on HBASE-19852:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
16s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
18s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
19s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m
28s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
54s{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19852 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915348/HBASE-19852.master.006.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile findbugs hbaseanti checkstyle |
| uname | Linux eb3ab13f1e1c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2a3f4a0a4e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
|
[jira] [Commented] (HBASE-19441) Implement retry logic around starting exclusive backup operation
[
https://issues.apache.org/jira/browse/HBASE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406687#comment-16406687
]
Josh Elser commented on HBASE-19441:
{code:java}
[WARNING]
/testptch/hbase/hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupManager.java:[87,19]
[MissingOverride] run implements method in Runnable; expected @Override
[WARNING]
/testptch/hbase/hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupManager.java:[120,24]
[ThreadJoinLoop] Thread.join needs to be surrounded by a loop until it
succeeds, as in Uninterruptibles.joinUninterruptibly.{code}
Also please fix the javac warnings
{code:java}
+ throw new IOException();
...
+} catch (IOException | InterruptedException e) {
+ fail("Unexpected exception: "+ e.getMessage());{code}
Can you please add a message to the IOException thrown in the test? If this
test would fail for some reason, we'd get a really non-intuitive explanation.
{noformat}
+assertTrue(startTimes.get(1) - startTimes.get(0) >= sleepTime);
+assertTrue(stopTimes.get(1) - stopTimes.get(0) >= sleepTime);{noformat}
Would be nice to print out the {{startTimes}} and {{stopTimes}} in the
respective messages for these assertions.
Otherwise, this looks fine to me. A nice improvement over what is presently in
master. Good work, Vlad.
> Implement retry logic around starting exclusive backup operation
>
>
> Key: HBASE-19441
> URL: https://issues.apache.org/jira/browse/HBASE-19441
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-19441-v1.patch, HBASE-19441-v2.patch
>
>
> {quote}
> Specifically, the client does a checkAndPut to specifics coordinates in the
> backup table and throws an exception when that fails. Remember that backups
> are client driven (per some design review from a long time ago), so queuing
> is tough to reason about (we have no "centralized" execution system to use).
> At a glance, it seems pretty straightforward to add some retry/backoff
> semantics to BackupSystemTable#startBackupExclusiveOperation().
> {quote}
> While we are in a state in which backup operations cannot be executed in
> parallel, it would be nice to provide some retry logic + configuration. This
> would alleviate users from having to build this themselves.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20231) Not able to delete column family from a row using RemoteHTable
[
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406684#comment-16406684
]
Ashish Singhi commented on HBASE-20231:
---
+1
> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-20231-branch-1.patch, HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:"
> instead of "/t1/r2/cf1".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406670#comment-16406670
]
stack edited comment on HBASE-19639 at 3/20/18 4:53 PM:
bq. didn't see any RegionTooBusyException...
[~eshcar] Yeah, we don't put them in the LOG... we just throw them to the
client. I see them in the MapReduce task logs, not in hbase regionserver logs.
Let me see if I can put up a metric or log message so we we can see when a
Region is struggling...
bq. and most flushes are when the memstore size is ~128MB, these are good news
I guess.
Yes. This is good.
Let me fix the log line.
bq. Anyway, I would strongly suggest to compare performance to a setting where
G1GC is used and no MSLAB.
Hmm. This is another project, having hbase2 do G1GC by default. And MSLAB has
been enabled always. You folks think it should be off ([~anastas] you were
looking at this?) by default? It might make sense when G1GC is enabled but
proving this generally would be a bit of a project.
was (Author: stack):
bq. didn't see any RegionTooBusyException...
[~eshcar] Yeah, we don't put them in the LOG... we just throw them to the
client. I see them in the MapReduce task logs, not in hbase regionserver logs.
Let me see if I can put up a metric or log message so we we can see when a
Region is struggling...
bq. and most flushes are when the memstore size is ~128MB, these are good news
I guess.
Yes. This is good.
Let me fix the log line.
bq. Anyway, I would strongly suggest to compare performance to a setting where
G1GC is used and no MSLAB.
Hmm. This is another project, having hbase2 do G1GC by default. And MSLAB has
bee
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
>
[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad
[
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406674#comment-16406674
]
Hadoop QA commented on HBASE-15291:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
30s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} hbase-server: The patch generated 0 new + 3
unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
27s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 4s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}180m 28s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}226m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestFromClientSide3 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-15291 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915314/HBASE-15291.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c7809d1d3db1 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 2a3f4a0a4e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/12037/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-19639) ITBLL can't go big because RegionTooBusyException... Above memstore limit
[
https://issues.apache.org/jira/browse/HBASE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406670#comment-16406670
]
stack commented on HBASE-19639:
---
bq. didn't see any RegionTooBusyException...
[~eshcar] Yeah, we don't put them in the LOG... we just throw them to the
client. I see them in the MapReduce task logs, not in hbase regionserver logs.
Let me see if I can put up a metric or log message so we we can see when a
Region is struggling...
bq. and most flushes are when the memstore size is ~128MB, these are good news
I guess.
Yes. This is good.
Let me fix the log line.
bq. Anyway, I would strongly suggest to compare performance to a setting where
G1GC is used and no MSLAB.
Hmm. This is another project, having hbase2 do G1GC by default. And MSLAB has
bee
> ITBLL can't go big because RegionTooBusyException... Above memstore limit
> -
>
> Key: HBASE-19639
> URL: https://issues.apache.org/jira/browse/HBASE-19639
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: hbase-stack-regionserver-ve0528.log.gz
>
>
> Running ITBLLs, the basic link generator keeps failing because I run into
> exceptions like below:
> {code}
> 2017-12-26 19:23:45,284 INFO [main]
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator:
> Persisting current.length=100, count=100, id=Job:
> job_1513025868268_0062 Task: attempt_1513025868268_0062_m_06_2,
> current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=100
> 2017-12-26 19:24:18,982 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null,
> retrying after=10050ms, replay=524ops
> 2017-12-26 19:24:29,061 INFO [htable-pool3-t6]
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2,
> table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last
> exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b.,
> server=ve0538.halxg.cloudera.com,16020,1514343549993,
> memstoreSize=538084641, blockingMemStoreSize=536870912
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
>
[jira] [Commented] (HBASE-20231) Not able to delete column family from a row using RemoteHTable
[
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406669#comment-16406669
]
Ted Yu commented on HBASE-20231:
[~chia7712]:
Can you also take a look ?
> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-20231-branch-1.patch, HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:"
> instead of "/t1/r2/cf1".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406664#comment-16406664
]
Ted Yu commented on HBASE-19343:
lgtm
You may want to re-attach the patch for good QA run.
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-19343-branch-1.patch, Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
>
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406650#comment-16406650
]
Kevin Risden commented on HBASE-19852:
--
reattached to try to get precommit to run.
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HBASE-19852:
-
Attachment: HBASE-19852.master.006.patch
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19343) Restore snapshot makes parent split region online
[
https://issues.apache.org/jira/browse/HBASE-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406643#comment-16406643
]
Pankaj Kumar commented on HBASE-19343:
--
Ping [~yuzhih...@gmail.com]. Please check this.
> Restore snapshot makes parent split region online
> --
>
> Key: HBASE-19343
> URL: https://issues.apache.org/jira/browse/HBASE-19343
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-19343-branch-1.patch, Snapshot.jpg
>
>
> Restore snapshot makes parent split region online as shown in the attached
> snapshot.
> Steps to reproduce
> =
> 1. Create table
> 2. Insert few records into the table
> 3. flush the table
> 4. Split the table
> 5. Create snapshot before catalog janitor clears the parent region entry from
> meta.
> 6. Restore snapshot
> We can see the problem in meta entries,
> Meta content before restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537565964, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> '', OFFLINE => true, SPLIT => true}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537530107,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537530107, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:serverstartcode, timestamp=1511537530107, value=1511537511523
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitA, timestamp=1511537565964, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY => '',
> ENDKEY => 'm'}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:splitB, timestamp=1511537565964, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY => 'm
> ', ENDKEY => ''}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:regioninfo, timestamp=1511537566075, value={ENCODED =>
> 3c7c866d4df370c586131a4cbe0ef6a8, NAME =>
> 't1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.', STARTKEY =>
> '', ENDKEY =>
> 'm'}
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:seqnumDuringOpen, timestamp=1511537566075,
> value=\x00\x00\x00\x00\x00\x00\x00\x02
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:server, timestamp=1511537566075, value=host-xx:16020
> t1,,1511537565718.3c7c866d4df370c586131a4cbe0ef6a8.
> column=info:serverstartcode, timestamp=1511537566075, value=1511537511523
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:regioninfo, timestamp=1511537566069, value={ENCODED =>
> dc7facd824c85b94e5bf6a2e6b5f5efc, NAME =>
> 't1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.', STARTKEY =
> > 'm', ENDKEY =>
> ''}
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:seqnumDuringOpen, timestamp=1511537566069,
> value=\x00\x00\x00\x00\x00\x00\x00\x08
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:server, timestamp=1511537566069, value=host-xx:16020
> t1,m,1511537565718.dc7facd824c85b94e5bf6a2e6b5f5efc.
> column=info:serverstartcode, timestamp=1511537566069, value=1511537511523
> {noformat}
> Meta content after restore snapshot:
> {noformat}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:regioninfo, timestamp=1511537667635, value={ENCODED =>
> 077a12b0b3c91b053fa95223635f9543, NAME =>
> 't1,,1511537529449.077a12b0b3c91b053fa95223635f9543.', STARTKEY =>
> '', ENDKEY =>
> ''}
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:seqnumDuringOpen, timestamp=1511537667635,
> value=\x00\x00\x00\x00\x00\x00\x00\x0A
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
> column=info:server, timestamp=1511537667635, value=host-xx:16020
> t1,,1511537529449.077a12b0b3c91b053fa95223635f9543.
>
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HBASE-19852:
-
Attachment: (was: HBASE-19852.master.005.patch)
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20231) Not able to delete column family from a row using RemoteHTable
[
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406631#comment-16406631
]
Pankaj Kumar commented on HBASE-20231:
--
*HBASE-20231-branch-1.patch* failures are not relevant.
They failed while resolving the dependencies,
{code:java}
Plugin org.apache.maven.plugins:maven-compiler-plugin:3.6.1 or one of its
dependencies could not be resolved
{code}
> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-20231-branch-1.patch, HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:"
> instead of "/t1/r2/cf1".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406621#comment-16406621 ] Ashish Singhi commented on HBASE-15291: --- [~yuzhih...@gmail.com], now there is no sharing of FS, each region has its own FS. In the patch we are using FileSystem#newInstance it will ensure each CF and region has its own FS. > FileSystem not closed in secure bulkLoad > > > Key: HBASE-15291 > URL: https://issues.apache.org/jira/browse/HBASE-15291 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.2, 0.98.16.1 >Reporter: Yong Zhang >Assignee: Ashish Singhi >Priority: Major > Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, > HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, > HBASE-15291.addendum, HBASE-15291.patch, patch > > > FileSystem not closed in secure bulkLoad after bulkLoad finish, it will > cause memory used more and more if too many bulkLoad . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HBASE-19852:
-
Status: Patch Available (was: Open)
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.005.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HBASE-19852:
-
Status: Open (was: Patch Available)
Cancelling to resubmit to get build to run.
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.005.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406572#comment-16406572 ] stack commented on HBASE-20188: --- [~anastas] yeah, lets expose metrics so we can 'watch' the in-memory compactions across time. File an issue? > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406571#comment-16406571 ] stack commented on HBASE-20188: --- I have defaults only. Nothing special. Let me know if I need to change anything and I'll change the defaults. Even my JVM is 'default': CMS. This test, ITBLL, is also a 'default' test. It is kinda like YCSB, something folks will just want to run. Only specialization are 31G heap and 48 handlers (I have 48 CPUs). > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: flamegraph-1072.1.svg, flamegraph-1072.2.svg, tree.txt > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[ https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406537#comment-16406537 ] Ted Yu commented on HBASE-20212: lgtm > Make all Public classes have InterfaceAudience category > --- > > Key: HBASE-20212 > URL: https://issues.apache.org/jira/browse/HBASE-20212 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.1.0 > > Attachments: HBASE-20212.v0.patch > > > The tasks will be resolved are shown below. > # add warbucks-maven-plugin to root pom > # make sure all sub modules ref the warbucks-maven-plugin > # remove old checker (TestInterfaceAudienceAnnotations) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406526#comment-16406526 ] Ted Yu commented on HBASE-15291: Can you outline how the race condition described earlier is addressed ? https://issues.apache.org/jira/browse/HBASE-15291?focusedCommentId=15371299=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15371299 > FileSystem not closed in secure bulkLoad > > > Key: HBASE-15291 > URL: https://issues.apache.org/jira/browse/HBASE-15291 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.2, 0.98.16.1 >Reporter: Yong Zhang >Assignee: Ashish Singhi >Priority: Major > Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, > HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, > HBASE-15291.addendum, HBASE-15291.patch, patch > > > FileSystem not closed in secure bulkLoad after bulkLoad finish, it will > cause memory used more and more if too many bulkLoad . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17449) Add explicit document on different timeout settings
[ https://issues.apache.org/jira/browse/HBASE-17449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406504#comment-16406504 ] stack commented on HBASE-17449: --- Thanks [~appy] I think this an important one. Me and Peter been hitting our heads off it. > Add explicit document on different timeout settings > --- > > Key: HBASE-17449 > URL: https://issues.apache.org/jira/browse/HBASE-17449 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Yu Li >Assignee: Yu Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: 0001-Standalone.patch, HBASE-17449.master.002.patch, > WIP-HBASE-17499.master.001.patch > > > Currently we have more than one timeout settings, mainly includes: > * hbase.rpc.timeout > * hbase.client.operation.timeout > * hbase.client.scanner.timeout.period > And in latest branch-1 or master branch code, we will have two other > properties: > * hbase.rpc.read.timeout > * hbase.rpc.write.timeout > However, in current refguid we don't have explicit instruction on the > difference of these timeout settings (there're explanations for each > property, but no instruction on when to use which) > In my understanding, for RPC layer timeout, or say each rpc call: > * Scan (openScanner/next): controlled by hbase.client.scanner.timeout.period > * Other operations: >1. For released versions: controlled by hbase.rpc.timeout >2. For 1.4+ versions: read operation controlled by hbase.rpc.read.timeout, > write operation controlled by hbase.rpc.write.timeout, or hbase.rpc.timeout > if the previous two are not set. > And hbase.client.operation.timeout is a higher-level control counting retry > in, or say the overall control for one user call. > After this JIRA, I hope when users ask questions like "What settings I should > use if I don't want to wait for more than 1 second for a single > put/get/scan.next call", we could give a neat answer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20169:
--
Summary: NPE when calling HBTU.shutdownMiniCluster
(TestAssignmentManagerMetrics is flakey) (was: NPE when calling
HBTU.shutdownMiniCluster)
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch,
> HBASE-20169.branch-2.004.patch, HBASE-20169.branch-2.005.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406497#comment-16406497
]
stack commented on HBASE-20169:
---
.005 fix checkstyle and the failed test was because ClassRule referred to wrong
class (copy/paste).
> NPE when calling HBTU.shutdownMiniCluster
> -
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch,
> HBASE-20169.branch-2.004.patch, HBASE-20169.branch-2.005.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20169:
--
Attachment: HBASE-20169.branch-2.005.patch
> NPE when calling HBTU.shutdownMiniCluster
> -
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch,
> HBASE-20169.branch-2.004.patch, HBASE-20169.branch-2.005.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406487#comment-16406487
]
stack commented on HBASE-20169:
---
bq. I like returning no more state a lot more than throwing an exception here!
[~mdrob] FYI, the above change came from having to add the 'uninteresting' unit
test. Thanks for the review.
> NPE when calling HBTU.shutdownMiniCluster
> -
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch,
> HBASE-20169.branch-2.004.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20226) Performance Improvement Taking Large Snapshots In Remote Filesystems
[
https://issues.apache.org/jira/browse/HBASE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406488#comment-16406488
]
Ted Yu commented on HBASE-20226:
{code}
+if (v1Regions.size() > 0 || v2Regions.size() > 0) {
{code}
I think you may tighten the above condition by checking the sum of the sizes.
{code}
+ ThreadPoolExecutor tpoolDelete =
createExecutor("SnapshotRegionManifestDeletePool");
{code}
where:
{code}
public static ThreadPoolExecutor createExecutor(final Configuration conf,
final String name) {
int maxThreads = conf.getInt("hbase.snapshot.thread.pool.max", 8);
{code}
You can add new config, instead of depending on the existing config above.
> Performance Improvement Taking Large Snapshots In Remote Filesystems
>
>
> Key: HBASE-20226
> URL: https://issues.apache.org/jira/browse/HBASE-20226
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 1.4.0
> Environment: HBase 1.4.0 running on an AWS EMR cluster with the
> hbase.rootdir set to point to a folder in S3
>Reporter: Saad Mufti
>Priority: Minor
> Attachments: HBASE-20226..01.patch
>
>
> When taking a snapshot of any table, one of the last steps is to delete the
> region manifests, which have already been rolled up into a larger overall
> manifest and thus have redundant information.
> This proposal is to do the deletion in a thread pool bounded by
> hbase.snapshot.thread.pool.max . For large tables with a lot of regions, the
> current single threaded deletion is taking longer than all the rest of the
> snapshot tasks when the Hbase data and the snapshot folder are both in a
> remote filesystem like S3.
> I have a patch for this proposal almost ready and will submit it tomorrow for
> feedback, although I haven't had a chance to write any tests yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20226) Performance Improvement Taking Large Snapshots In Remote Filesystems
[
https://issues.apache.org/jira/browse/HBASE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406469#comment-16406469
]
Ted Yu commented on HBASE-20226:
The following is reproducible :
{code}
testReadSnapshotManifest(org.apache.hadoop.hbase.snapshot.TestSnapshotManifest)
Time elapsed: 0.064 sec <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.hbase.snapshot.TestSnapshotManifest.setup(TestSnapshotManifest.java:83)
{code}
See if fixing FindBugs warnings would avoid the above exception.
> Performance Improvement Taking Large Snapshots In Remote Filesystems
>
>
> Key: HBASE-20226
> URL: https://issues.apache.org/jira/browse/HBASE-20226
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 1.4.0
> Environment: HBase 1.4.0 running on an AWS EMR cluster with the
> hbase.rootdir set to point to a folder in S3
>Reporter: Saad Mufti
>Priority: Minor
> Attachments: HBASE-20226..01.patch
>
>
> When taking a snapshot of any table, one of the last steps is to delete the
> region manifests, which have already been rolled up into a larger overall
> manifest and thus have redundant information.
> This proposal is to do the deletion in a thread pool bounded by
> hbase.snapshot.thread.pool.max . For large tables with a lot of regions, the
> current single threaded deletion is taking longer than all the rest of the
> snapshot tasks when the Hbase data and the snapshot folder are both in a
> remote filesystem like S3.
> I have a patch for this proposal almost ready and will submit it tomorrow for
> feedback, although I haven't had a chance to write any tests yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20231) Not able to delete column family from a row using RemoteHTable
[
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406462#comment-16406462
]
Ted Yu commented on HBASE-20231:
HBASE-20231.patch looks good to me.
> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-20231-branch-1.patch, HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:"
> instead of "/t1/r2/cf1".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20212) Make all Public classes have InterfaceAudience category
[
https://issues.apache.org/jira/browse/HBASE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406459#comment-16406459
]
Hadoop QA commented on HBASE-20212:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 13m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hbase-annotations hbase-spark-it . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 6m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch hbase-annotations passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} hbase-common: The patch generated 0 new + 12
unchanged - 2 fixed = 12 total (was 14) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch hbase-metrics-api passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch hbase-hadoop-compat passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch hbase-metrics passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hbase-hadoop2-compat: The patch generated 0 new + 6
unchanged - 3 fixed = 6 total (was 9) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch hbase-protocol passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} The patch hbase-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hbase-zookeeper passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hbase-replication passed checkstyle
{color} |
|
