[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-830540794 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 42s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 27s | master passed | | +1 :green_heart: | compile | 2m 56s | master passed | | +1 :green_heart: | shadedjars | 9m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 21s | the patch passed | | +1 :green_heart: | compile | 2m 53s | the patch passed | | +1 :green_heart: | javac | 2m 53s | the patch passed | | +1 :green_heart: | shadedjars | 8m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 447m 13s | root in the patch passed. | | | | 490m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1e3bed649845 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5d42f58ff6 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/testReport/ | | Max. process+thread count | 3915 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25825) RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer
[
https://issues.apache.org/jira/browse/HBASE-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337711#comment-17337711
]
Hudson commented on HBASE-25825:
Results for branch branch-2.4
[build #108 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/108/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/108/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/108/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/108/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/108/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to
> internal balancer
>
>
> Key: HBASE-25825
> URL: https://issues.apache.org/jira/browse/HBASE-25825
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3, 2.3.6
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-830487634 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 17s | master passed | | +1 :green_heart: | compile | 3m 30s | master passed | | +1 :green_heart: | shadedjars | 9m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 4s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 0s | the patch passed | | +1 :green_heart: | compile | 3m 26s | the patch passed | | +1 :green_heart: | javac | 3m 26s | the patch passed | | +1 :green_heart: | shadedjars | 9m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 28s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 245m 25s | root in the patch failed. | | | | 294m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 996d046a9391 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5d42f58ff6 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/testReport/ | | Max. process+thread count | 3139 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25832) Performance improvements for StochasticLoadBalancer
Duo Zhang created HBASE-25832:
-
Summary: Performance improvements for StochasticLoadBalancer
Key: HBASE-25832
URL: https://issues.apache.org/jira/browse/HBASE-25832
Project: HBase
Issue Type: Improvement
Components: Balancer, Performance
Reporter: Duo Zhang
This is what I posted in the slack channel.
{quote}
When refactoring balancer related code for moving them out to hbase-balancer
module, I found that we will create a OffPeakHours instance in
MoveCostFunction.cost every time. So I wrote a simple JMH test to test its
performance.
https://github.com/Apache9/hbase-jmh-test/blob/master/src/main/java/org/apache/hadoop/hbase/master/balancer/MoveCostFunctionBenchmark.java
For the original version, the result is
BenchmarkMode SamplesScore
Score error Units
o.a.h.h.m.b.MoveCostFunctionBenchmark.computeCostavgt 10 242.312
3.853 ns/op
12:06
If I initialized OffPeakHours in constructor, then the result is
BenchmarkMode Samples Score
Score error Units
o.a.h.h.m.b.MoveCostFunctionBenchmark.computeCostavgt 10 81.186
0.720 ns/op
12:06
If I also eliminate the conf.getFloat in the cost method(by storing them to a
field member in constructor, the result is
BenchmarkMode Samples Score
Score error Units
o.a.h.h.m.b.MoveCostFunctionBenchmark.computeCostavgt 10 4.419
0.035 ns/op
12:08
I think on a large cluster, the performance for balancer is a problem, so maybe
we should try to avoid creating too many objects or calling conf.getXXX in the
critical path? As we will call StochasticLoadBalancer.computeCost in every step
when balancing cluster...
{quote}
For large clusters, the performance of StochasticLoadBalancer will be very
important, so I think we need to treat this problem seriously and try to
improve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell reassigned HBASE-25829:
---
Assignee: Andrew Kyle Purtell
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
> {noformat}
> The HBCK chore notices them but does nothing:
> "Loaded *80 regions* from in-memory state of AssignmentManager"
> "Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
> "Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
> Yes, there are exactly 7 region state records of SPLIT state with
> server=null.
> {noformat}
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
> orphan regions
> 2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0
> orphan regions
> {noformat}
> This repeats indefinitely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337690#comment-17337690
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 11:41 PM:
This addresses the issue of multiple split request transaction submissions.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
index 107330d90b..48cc26086f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
@@ -1119,7 +1119,13 @@ public class AssignmentManager {
LOG.debug("Split request from " + serverName +
", parent=" + parent + " splitKey=" +
Bytes.toStringBinary(splitKey));
}
-
master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent,
splitKey));
+if (regionStates.getRegionState(parent).isOpened() &&
+ !regionStates.getRegionState(parent).isSplitting()) {
+
master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent,
splitKey));
+} else {
+ LOG.warn("Ignoring split request from " + serverName +
+", parent=" + parent + " because parent is already splitting or not
online");
+}
// If the RS is < 2.0 throw an exception to abort the operation, we are
handling the split
if (master.getServerManager().getVersionNumber(serverName) < 0x020) {
{code}
Is this the complete fix, though?
With this patch in place, now we just have in the master log:
{noformat}
2021-04-30 23:22:14,971 WARN
[RpcServer.priority.RWQ.Codel.write.handler=1,queue=0,port=8100]
assignment.AssignmentManager: Ignoring split request from
ip-172-31-63-83.us-west-2.compute.internal,8120,1619824775800, parent={ENCODED
=> df7aa0e0af5a2b757ad86f2cf051fcbb, NAME =>
'IntegrationTestLoadCommonCrawl,,1619824793285.df7aa0e0af5a2b757ad86f2cf051fcbb.',
STARTKEY => '', ENDKEY => ''} because parent is already splitting or not online
{noformat}
but the first submission of the split request is already in progress and
completes just fine.
was (Author: apurtell):
This addresses the issue of multiple split request transaction submissions.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
index 107330d90b..48cc26086f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
@@ -1119,7 +1119,13 @@ public class AssignmentManager {
LOG.debug("Split request from " + serverName +
", parent=" + parent + " splitKey=" +
Bytes.toStringBinary(splitKey));
}
-
master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent,
splitKey));
+if (regionStates.getRegionState(parent).isOpened() &&
+ !regionStates.getRegionState(parent).isSplitting()) {
+
master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent,
splitKey));
+} else {
+ LOG.warn("Ignoring split request from " + serverName +
+", parent=" + parent + " because parent is already splitting or not
online");
+}
// If the RS is < 2.0 throw an exception to abort the operation, we are
handling the split
if (master.getServerManager().getVersionNumber(serverName) < 0x020) {
{code}
Is this the complete fix, though? Should the RS be submitting this report more
than once?
With this patch in place, now we just have in the master log:
{noformat}
2021-04-30 23:22:14,971 WARN
[RpcServer.priority.RWQ.Codel.write.handler=1,queue=0,port=8100]
assignment.AssignmentManager: Ignoring split request from
ip-172-31-63-83.us-west-2.compute.internal,8120,1619824775800, parent={ENCODED
=> df7aa0e0af5a2b757ad86f2cf051fcbb, NAME =>
'IntegrationTestLoadCommonCrawl,,1619824793285.df7aa0e0af5a2b757ad86f2cf051fcbb.',
STARTKEY => '', ENDKEY => ''} because parent is already splitting or not online
{noformat}
but the first submission of the split request is already in progress and
completes just fine.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337694#comment-17337694
]
Andrew Kyle Purtell commented on HBASE-25829:
-
So we have two issues here that I can submit PRs for in subtasks:
1. RegionStates#getAssignmentsForBalancer should only care about OPEN or
OPENING regions
2. Ignore duplicate split requests from regionserver reports.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
> {noformat}
> The HBCK chore notices them but does nothing:
> "Loaded *80 regions* from in-memory state of AssignmentManager"
> "Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
> "Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
> Yes, there are exactly 7 region state records of SPLIT state with
> server=null.
> {noformat}
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
> orphan regions
> 2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0
> orphan regions
> {noformat}
> This repeats indefinitely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337690#comment-17337690
]
Andrew Kyle Purtell commented on HBASE-25829:
-
This addresses the issue of multiple split request transaction submissions.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
index 107330d90b..48cc26086f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
@@ -1119,7 +1119,13 @@ public class AssignmentManager {
LOG.debug("Split request from " + serverName +
", parent=" + parent + " splitKey=" +
Bytes.toStringBinary(splitKey));
}
-
master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent,
splitKey));
+if (regionStates.getRegionState(parent).isOpened() &&
+ !regionStates.getRegionState(parent).isSplitting()) {
+
master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent,
splitKey));
+} else {
+ LOG.warn("Ignoring split request from " + serverName +
+", parent=" + parent + " because parent is already splitting or not
online");
+}
// If the RS is < 2.0 throw an exception to abort the operation, we are
handling the split
if (master.getServerManager().getVersionNumber(serverName) < 0x020) {
{code}
Is this the complete fix, though? Should the RS be submitting this report more
than once?
With this patch in place, now we just have in the master log:
{noformat}
2021-04-30 23:22:14,971 WARN
[RpcServer.priority.RWQ.Codel.write.handler=1,queue=0,port=8100]
assignment.AssignmentManager: Ignoring split request from
ip-172-31-63-83.us-west-2.compute.internal,8120,1619824775800, parent={ENCODED
=> df7aa0e0af5a2b757ad86f2cf051fcbb, NAME =>
'IntegrationTestLoadCommonCrawl,,1619824793285.df7aa0e0af5a2b757ad86f2cf051fcbb.',
STARTKEY => '', ENDKEY => ''} because parent is already splitting or not online
{noformat}
but the first submission of the split request is already in progress and
completes just fine.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
>
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337654#comment-17337654
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 10:55 PM:
{quote}In RegionStates#getAssignmentsForBalancer, we have a check for
isTableDisabled or isSplitParent. Both exclude some subset of regions that
should not be candidates for consideration, but why not get to the point?
{quote}
The change is good on its own merits, but I don't believe the balancer is doing
something it should not, so won't address the root issue. I will follow up on
getAssignmentsForBalancer change in a subtask.
Back to the split state handling. Perhaps we are trying to create a
SplitTransitionProcedure for the split more than once.
was (Author: apurtell):
{quote}In RegionStates#getAssignmentsForBalancer, we have a check for
isTableDisabled or isSplitParent. Both exclude some subset of regions that
should not be candidates for consideration, but why not get to the point?
{quote}
This helps, but it treats the symptom. I will follow up on this in a subtask.
Back to the split state handling. Perhaps we are trying to create a
SplitTransitionProcedure for the split more than once.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
> {noformat}
> The HBCK chore notices them but does nothing:
> "Loaded *80 regions* from in-memory state of AssignmentManager"
> "Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
> "Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
> Yes, there are exactly 7 region state records of SPLIT state with
> server=null.
> {noformat}
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
> orphan regions
> 2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
>
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337654#comment-17337654
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 10:50 PM:
{quote}In RegionStates#getAssignmentsForBalancer, we have a check for
isTableDisabled or isSplitParent. Both exclude some subset of regions that
should not be candidates for consideration, but why not get to the point?
{quote}
This helps, but it treats the symptom. I will follow up on this in a subtask.
Back to the split state handling. Perhaps we are trying to create a
SplitTransitionProcedure for the split more than once.
was (Author: apurtell):
bq. In RegionStates#getAssignmentsForBalancer, we have a check for
isTableDisabled or isSplitParent. Both exclude some subset of regions that
should not be candidates for consideration, but why not get to the point?
This helps, but it treats the symptom. I will follow up on this in a subtask.
Back to the split state handling. Why is this printed three times?
{noformat}
2021-04-30 22:06:15,686 INFO [PEWorker-6]
procedure.MasterProcedureScheduler: Took xlock for pid=93,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=2d64bc9357b5f780307611f43052e092,
daughterA=817c9985e6a2ee7645d3a292758ec729,
daughterB=8ea34c3e5bcfb4c7d81e4212d700b824
2021-04-30 22:06:15,686 INFO [PEWorker-6]
procedure.MasterProcedureScheduler: Took xlock for pid=93,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=2d64bc9357b5f780307611f43052e092,
daughterA=817c9985e6a2ee7645d3a292758ec729,
daughterB=8ea34c3e5bcfb4c7d81e4212d700b824
2021-04-30 22:06:15,686 INFO [PEWorker-6]
procedure.MasterProcedureScheduler: Took xlock for pid=93,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=2d64bc9357b5f780307611f43052e092,
daughterA=817c9985e6a2ee7645d3a292758ec729,
daughterB=8ea34c3e5bcfb4c7d81e4212d700b824
{noformat}
because what follows is
1. A successful split.
2. A failed SplitTableRegionProcedure with assignment.AssignmentManager: Failed
transition org.apache.hadoop.hbase.client.DoNotRetryRegionException:
2d64bc9357b5f780307611f43052e092 is not OPEN; state=CLOSING
3. Another failed SplitTableRegionProcedure with assignment.AssignmentManager:
Failed transition org.apache.hadoop.hbase.client.DoNotRetryRegionException:
2d64bc9357b5f780307611f43052e092 is not OPEN; state=CLOSING
So we ran the split three times? And only one succeeded.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
>
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-830429718 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 8m 13s | master passed | | +1 :green_heart: | checkstyle | 1m 55s | master passed | | +1 :green_heart: | spotbugs | 9m 50s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 8m 17s | the patch passed | | +1 :green_heart: | javac | 8m 17s | the patch passed | | +1 :green_heart: | checkstyle | 1m 55s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 17m 52s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 10m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 75m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 97359cc2ba2b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5d42f58ff6 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 141 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337654#comment-17337654
]
Andrew Kyle Purtell commented on HBASE-25829:
-
bq. In RegionStates#getAssignmentsForBalancer, we have a check for
isTableDisabled or isSplitParent. Both exclude some subset of regions that
should not be candidates for consideration, but why not get to the point?
This helps, but it treats the symptom. I will follow up on this in a subtask.
Back to the split state handling. Why is this printed three times?
{noformat}
2021-04-30 22:06:15,686 INFO [PEWorker-6]
procedure.MasterProcedureScheduler: Took xlock for pid=93,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=2d64bc9357b5f780307611f43052e092,
daughterA=817c9985e6a2ee7645d3a292758ec729,
daughterB=8ea34c3e5bcfb4c7d81e4212d700b824
2021-04-30 22:06:15,686 INFO [PEWorker-6]
procedure.MasterProcedureScheduler: Took xlock for pid=93,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=2d64bc9357b5f780307611f43052e092,
daughterA=817c9985e6a2ee7645d3a292758ec729,
daughterB=8ea34c3e5bcfb4c7d81e4212d700b824
2021-04-30 22:06:15,686 INFO [PEWorker-6]
procedure.MasterProcedureScheduler: Took xlock for pid=93,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=2d64bc9357b5f780307611f43052e092,
daughterA=817c9985e6a2ee7645d3a292758ec729,
daughterB=8ea34c3e5bcfb4c7d81e4212d700b824
{noformat}
because what follows is
1. A successful split.
2. A failed SplitTableRegionProcedure with assignment.AssignmentManager: Failed
transition org.apache.hadoop.hbase.client.DoNotRetryRegionException:
2d64bc9357b5f780307611f43052e092 is not OPEN; state=CLOSING
3. Another failed SplitTableRegionProcedure with assignment.AssignmentManager:
Failed transition org.apache.hadoop.hbase.client.DoNotRetryRegionException:
2d64bc9357b5f780307611f43052e092 is not OPEN; state=CLOSING
So we ran the split three times? And only one succeeded.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337628#comment-17337628
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 9:25 PM:
---
I'm going to try out a patch, see if it helps. In theory this both excludes
regions in SPLITTING or SPLITTING_NEW from balancer attempts to mutate
assignments and skips a bunch of other uninteresting (as in not actively
assigned) states, reducing balancer workload and also avoiding unintended
consequences.
In RegionStates#getAssignmentsForBalancer, we have a check for isTableDisabled
or isSplitParent. Both exclude some subset of regions that should not be
candidates for consideration, but why not get to the point?
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
index d3553f11a3..64abd4e3ac 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
@@ -553,23 +552,25 @@ public class RegionStates {
* wants to iterate this exported list. We need to synchronize on regions
* since all access to this.servers is under a lock on this.regions.
*
- * @return A clone of current assignments.
+ * @return A clone of current open or opening assignments.
*/
public Map>>
getAssignmentsForBalancer(
TableStateManager tableStateManager, List onlineServers) {
final Map>> result = new
HashMap<>();
for (RegionStateNode node : regionsMap.values()) {
- if (isTableDisabled(tableStateManager, node.getTable())) {
-continue;
- }
- if (node.getRegionInfo().isSplitParent()) {
+ // When balancing, we are only interested in OPEN or OPENING regions and
expected
+ // to be online at that server until possibly the next balancer
iteration or unless
+ // we decide to move it. Other states are not interesting as the region
will either
+ // be closing, or splitting/merging, or will not be deployed.
+ if (!(node.isInState(State.OPEN)||node.isInState(State.OPENING))) {
continue;
}
Map> tableResult =
result.computeIfAbsent(node.getTable(), t -> new HashMap<>());
final ServerName serverName = node.getRegionLocation();
+ // A region in ONLINE or OPENING state should have a location.
if (serverName == null) {
-LOG.info("Skipping, no server for " + node);
+LOG.warn("Skipping, no server for " + node);
continue;
}
List serverResult =
{code}
was (Author: apurtell):
I'm going to try out a patch, see if it helps.
In RegionStates#getAssignmentsForBalancer, we have a check for isTableDisabled
or isSplitParent. Both exclude some subset of regions that should not be
candidates for consideration, but why not get to the point?
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
index d3553f11a3..64abd4e3ac 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
@@ -553,23 +552,25 @@ public class RegionStates {
* wants to iterate this exported list. We need to synchronize on regions
* since all access to this.servers is under a lock on this.regions.
*
- * @return A clone of current assignments.
+ * @return A clone of current open or opening assignments.
*/
public Map>>
getAssignmentsForBalancer(
TableStateManager tableStateManager, List onlineServers) {
final Map>> result = new
HashMap<>();
for (RegionStateNode node : regionsMap.values()) {
- if (isTableDisabled(tableStateManager, node.getTable())) {
-continue;
- }
- if (node.getRegionInfo().isSplitParent()) {
+ // When balancing, we are only interested in OPEN or OPENING regions and
expected
+ // to be online at that server until possibly the next balancer
iteration or unless
+ // we decide to move it. Other states are not interesting as the region
will either
+ // be closing, or splitting/merging, or will not be deployed.
+ if (!(node.isInState(State.OPEN)||node.isInState(State.OPENING))) {
continue;
}
Map> tableResult =
result.computeIfAbsent(node.getTable(), t -> new HashMap<>());
final ServerName serverName = node.getRegionLocation();
+ // A region in ONLINE or OPENING state should have a location.
if (serverName == null) {
-LOG.info("Skipping, no server for " + node);
+LOG.warn("Skipping,
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337628#comment-17337628
]
Andrew Kyle Purtell commented on HBASE-25829:
-
I'm going to try out a patch, see if it helps.
In RegionStates#getAssignmentsForBalancer, we have a check for isTableDisabled
or isSplitParent. Both exclude some subset of regions that should not be
candidates for consideration, but why not get to the point?
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
index d3553f11a3..64abd4e3ac 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
@@ -553,23 +552,25 @@ public class RegionStates {
* wants to iterate this exported list. We need to synchronize on regions
* since all access to this.servers is under a lock on this.regions.
*
- * @return A clone of current assignments.
+ * @return A clone of current open or opening assignments.
*/
public Map>>
getAssignmentsForBalancer(
TableStateManager tableStateManager, List onlineServers) {
final Map>> result = new
HashMap<>();
for (RegionStateNode node : regionsMap.values()) {
- if (isTableDisabled(tableStateManager, node.getTable())) {
-continue;
- }
- if (node.getRegionInfo().isSplitParent()) {
+ // When balancing, we are only interested in OPEN or OPENING regions and
expected
+ // to be online at that server until possibly the next balancer
iteration or unless
+ // we decide to move it. Other states are not interesting as the region
will either
+ // be closing, or splitting/merging, or will not be deployed.
+ if (!(node.isInState(State.OPEN)||node.isInState(State.OPENING))) {
continue;
}
Map> tableResult =
result.computeIfAbsent(node.getTable(), t -> new HashMap<>());
final ServerName serverName = node.getRegionLocation();
+ // A region in ONLINE or OPENING state should have a location.
if (serverName == null) {
-LOG.info("Skipping, no server for " + node);
+LOG.warn("Skipping, no server for " + node);
continue;
}
List serverResult =
{code}
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO
[GitHub] [hbase] jatsakthi commented on pull request #3211: HBASE-25816: Improve the documentation of Architecture section of reference guide
jatsakthi commented on pull request #3211: URL: https://github.com/apache/hbase/pull/3211#issuecomment-830373287 Merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] jatsakthi merged pull request #3211: HBASE-25816: Improve the documentation of Architecture section of reference guide
jatsakthi merged pull request #3211: URL: https://github.com/apache/hbase/pull/3211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337591#comment-17337591
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 8:04 PM:
---
This warning repeats frequently and involves the split parents that accumulate
as bad state:
{noformat}
2021-04-30 19:35:01,115 INFO [PEWorker-27]
procedure.MasterProcedureScheduler: Took xlock for pid=313, ppid=312,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=c9076fb746a0690900130cd5d38f1815,
UNASSIGN
2021-04-30 19:35:01,116 INFO [PEWorker-27]
assignment.RegionStateStore: pid=313 updating hbase:meta
row=c9076fb746a0690900130cd5d38f1815, regionState=CLOSING,
regionLocation=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832
2021-04-30 19:35:01,116 INFO [PEWorker-27]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=314, ppid=313,
state=RUNNABLE; CloseRegionProcedure c9076fb746a0690900130cd5d38f1815,
server=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832}]
2021-04-30 19:35:05,546 WARN
[RpcServer.priority.RWQ.Codel.write.handler=0,queue=0,port=8100]
assignment.AssignmentManager: Failed transition
org.apache.hadoop.hbase.client.DoNotRetryRegionException:
c9076fb746a0690900130cd5d38f1815 is not OPEN; state=CLOSING
at
org.apache.hadoop.hbase.master.assignment.RegionStateNode.checkOnline(RegionStateNode.java:291)
at
org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:194)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:939)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:1122)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:984)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:1019)
at
org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1727)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:16185)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{noformat}
The "Skipping, no server for state=SPLIT" messages accumulate for the affected
split parents.
Not every split is affected.
In the master logs I see balancer and split activity co-occurring most of the
time. One may be interefering with the other. We seem to get into trouble only
when balancer activity and split procedure log lines are interlaced in the
master log.
was (Author: apurtell):
This warning repeats frequently and involves the split parents that accumulate
as bad state:
{noformat}
2021-04-30 19:35:01,115 INFO [PEWorker-27]
procedure.MasterProcedureScheduler: Took xlock for pid=313, ppid=312,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=c9076fb746a0690900130cd5d38f1815,
UNASSIGN
2021-04-30 19:35:01,116 INFO [PEWorker-27]
assignment.RegionStateStore: pid=313 updating hbase:meta
row=c9076fb746a0690900130cd5d38f1815, regionState=CLOSING,
regionLocation=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832
2021-04-30 19:35:01,116 INFO [PEWorker-27]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=314, ppid=313,
state=RUNNABLE; CloseRegionProcedure c9076fb746a0690900130cd5d38f1815,
server=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832}]
2021-04-30 19:35:05,546 WARN
[RpcServer.priority.RWQ.Codel.write.handler=0,queue=0,port=8100]
assignment.AssignmentManager: Failed transition
org.apache.hadoop.hbase.client.DoNotRetryRegionException:
c9076fb746a0690900130cd5d38f1815 is not OPEN; state=CLOSING
at
org.apache.hadoop.hbase.master.assignment.RegionStateNode.checkOnline(RegionStateNode.java:291)
at
org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:194)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
at
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337591#comment-17337591
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 8:00 PM:
---
This warning repeats frequently and involves the split parents that accumulate
as bad state:
{noformat}
2021-04-30 19:35:01,115 INFO [PEWorker-27]
procedure.MasterProcedureScheduler: Took xlock for pid=313, ppid=312,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=c9076fb746a0690900130cd5d38f1815,
UNASSIGN
2021-04-30 19:35:01,116 INFO [PEWorker-27]
assignment.RegionStateStore: pid=313 updating hbase:meta
row=c9076fb746a0690900130cd5d38f1815, regionState=CLOSING,
regionLocation=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832
2021-04-30 19:35:01,116 INFO [PEWorker-27]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=314, ppid=313,
state=RUNNABLE; CloseRegionProcedure c9076fb746a0690900130cd5d38f1815,
server=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832}]
2021-04-30 19:35:05,546 WARN
[RpcServer.priority.RWQ.Codel.write.handler=0,queue=0,port=8100]
assignment.AssignmentManager: Failed transition
org.apache.hadoop.hbase.client.DoNotRetryRegionException:
c9076fb746a0690900130cd5d38f1815 is not OPEN; state=CLOSING
at
org.apache.hadoop.hbase.master.assignment.RegionStateNode.checkOnline(RegionStateNode.java:291)
at
org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:194)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:939)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:1122)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:984)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:1019)
at
org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1727)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:16185)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{noformat}
The "Skipping, no server for state=SPLIT" messages accumulate for the affected
split parents.
Not every split is affected.
In the master logs I see balancer and split activity co-occurring most of the
time. One may be interefering with the other.
was (Author: apurtell):
This warning happens at every split:
{noformat}
2021-04-30 19:35:01,115 INFO [PEWorker-27]
procedure.MasterProcedureScheduler: Took xlock for pid=313, ppid=312,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=c9076fb746a0690900130cd5d38f1815,
UNASSIGN
2021-04-30 19:35:01,116 INFO [PEWorker-27]
assignment.RegionStateStore: pid=313 updating hbase:meta
row=c9076fb746a0690900130cd5d38f1815, regionState=CLOSING,
regionLocation=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832
2021-04-30 19:35:01,116 INFO [PEWorker-27]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=314, ppid=313,
state=RUNNABLE; CloseRegionProcedure c9076fb746a0690900130cd5d38f1815,
server=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832}]
2021-04-30 19:35:05,546 WARN
[RpcServer.priority.RWQ.Codel.write.handler=0,queue=0,port=8100]
assignment.AssignmentManager: Failed transition
org.apache.hadoop.hbase.client.DoNotRetryRegionException:
c9076fb746a0690900130cd5d38f1815 is not OPEN; state=CLOSING
at
org.apache.hadoop.hbase.master.assignment.RegionStateNode.checkOnline(RegionStateNode.java:291)
at
org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:194)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:939)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:1122)
at
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337591#comment-17337591
]
Andrew Kyle Purtell commented on HBASE-25829:
-
This warning happens at every split:
{noformat}
2021-04-30 19:35:01,115 INFO [PEWorker-27]
procedure.MasterProcedureScheduler: Took xlock for pid=313, ppid=312,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=c9076fb746a0690900130cd5d38f1815,
UNASSIGN
2021-04-30 19:35:01,116 INFO [PEWorker-27]
assignment.RegionStateStore: pid=313 updating hbase:meta
row=c9076fb746a0690900130cd5d38f1815, regionState=CLOSING,
regionLocation=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832
2021-04-30 19:35:01,116 INFO [PEWorker-27]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=314, ppid=313,
state=RUNNABLE; CloseRegionProcedure c9076fb746a0690900130cd5d38f1815,
server=ip-172-31-49-8.us-west-2.compute.internal,8120,1619809723832}]
2021-04-30 19:35:05,546 WARN
[RpcServer.priority.RWQ.Codel.write.handler=0,queue=0,port=8100]
assignment.AssignmentManager: Failed transition
org.apache.hadoop.hbase.client.DoNotRetryRegionException:
c9076fb746a0690900130cd5d38f1815 is not OPEN; state=CLOSING
at
org.apache.hadoop.hbase.master.assignment.RegionStateNode.checkOnline(RegionStateNode.java:291)
at
org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:194)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:939)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:1122)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:984)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:1019)
at
org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1727)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:16185)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{noformat}
And the "Skipping, no server for state=SPLIT" messages accumulate for every
split parent. If there are 10 splits in a test run, then every balancer
iteration we will log that line for 10 regions (the split parents).
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl,
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337586#comment-17337586
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 7:41 PM:
---
I have a nonstandard balancer configuration. This might be relevant:
hbase.balancer.period=6
hbase.master.balancer.stochastic.minCostNeedBalance=0.025
hbase.master.balancer.stochastic.regionCountCost=100
hbase.master.balancer.stochastic.localityCost=30
hbase.master.balancer.stochastic.moveCost=10
hbase.master.balancer.stochastic.readRequestCost=10
hbase.master.balancer.stochastic.writeRequestCost=15
I think there is a state desynchronization bug related to splitting. The weird
log lines reported here are for the split parent after a split completes.
Split procedure begins:
{noformat}
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
{noformat}
UNASSIGN parent region (22ab65a3e71c3ab7f94c1c47067ad54e), includes WARN for
failed transition:
{noformat}
2021-04-30 19:11:39,878 INFO [PEWorker-65]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=80, ppid=79,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=22ab65a3e71c3ab7f94c1c47067ad54e,
UNASSIGN}]
2021-04-30 19:11:39,878 INFO [PEWorker-66]
procedure.MasterProcedureScheduler: Took xlock for pid=80, ppid=79,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=22ab65a3e71c3ab7f94c1c47067ad54e,
UNASSIGN
2021-04-30 19:11:39,879 INFO [PEWorker-66]
assignment.RegionStateStore: pid=80 updating hbase:meta
row=22ab65a3e71c3ab7f94c1c47067ad54e, regionState=CLOSING,
regionLocation=ip-172-31-63-65.us-west-2.compute.internal,8120,1619809716877
2021-04-30 19:11:40,032 INFO [PEWorker-66]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=81, ppid=80,
state=RUNNABLE; CloseRegionProcedure 22ab65a3e71c3ab7f94c1c47067ad54e,
server=ip-172-31-63-65.us-west-2.compute.internal,8120,1619809716877}]
2021-04-30 19:11:40,303 WARN
[RpcServer.priority.RWQ.Codel.write.handler=0,queue=0,port=8100]
assignment.AssignmentManager: Failed transition
org.apache.hadoop.hbase.client.DoNotRetryRegionException:
22ab65a3e71c3ab7f94c1c47067ad54e is not OPEN; state=CLOSING
at
org.apache.hadoop.hbase.master.assignment.RegionStateNode.checkOnline(RegionStateNode.java:291)
at
org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:194)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:939)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:1122)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:984)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:1019)
at
org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1727)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:16185)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
2021-04-30 19:11:44,814 INFO [PEWorker-68]
assignment.RegionStateStore: pid=80 updating hbase:meta
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337586#comment-17337586
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 7:31 PM:
---
This is what happens.
I have a nonstandard balancer configuration. It is moving regions during a
split.
hbase.balancer.period=6
hbase.master.balancer.stochastic.minCostNeedBalance=0.025
hbase.master.balancer.stochastic.regionCountCost=100
hbase.master.balancer.stochastic.localityCost=30
hbase.master.balancer.stochastic.moveCost=10
hbase.master.balancer.stochastic.readRequestCost=10
hbase.master.balancer.stochastic.writeRequestCost=15
I think there is a state desynchronization bug in the master when the balancer
decides to move regions around when a split is in progress.
The weird log lines reported here are for the split parent after a split
completes.
Split procedure begins:
{noformat}
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
{noformat}
UNASSIGN parent region (22ab65a3e71c3ab7f94c1c47067ad54e), includes WARN for
failed transition:
{noformat}
2021-04-30 19:11:39,878 INFO [PEWorker-65]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=80, ppid=79,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=22ab65a3e71c3ab7f94c1c47067ad54e,
UNASSIGN}]
2021-04-30 19:11:39,878 INFO [PEWorker-66]
procedure.MasterProcedureScheduler: Took xlock for pid=80, ppid=79,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=22ab65a3e71c3ab7f94c1c47067ad54e,
UNASSIGN
2021-04-30 19:11:39,879 INFO [PEWorker-66]
assignment.RegionStateStore: pid=80 updating hbase:meta
row=22ab65a3e71c3ab7f94c1c47067ad54e, regionState=CLOSING,
regionLocation=ip-172-31-63-65.us-west-2.compute.internal,8120,1619809716877
2021-04-30 19:11:40,032 INFO [PEWorker-66]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=81, ppid=80,
state=RUNNABLE; CloseRegionProcedure 22ab65a3e71c3ab7f94c1c47067ad54e,
server=ip-172-31-63-65.us-west-2.compute.internal,8120,1619809716877}]
2021-04-30 19:11:40,303 WARN
[RpcServer.priority.RWQ.Codel.write.handler=0,queue=0,port=8100]
assignment.AssignmentManager: Failed transition
org.apache.hadoop.hbase.client.DoNotRetryRegionException:
22ab65a3e71c3ab7f94c1c47067ad54e is not OPEN; state=CLOSING
at
org.apache.hadoop.hbase.master.assignment.RegionStateNode.checkOnline(RegionStateNode.java:291)
at
org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:194)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:939)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:1122)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:984)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:1019)
at
org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1727)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:16185)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
2021-04-30
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337586#comment-17337586
]
Andrew Kyle Purtell commented on HBASE-25829:
-
This is what happens.
I have a nonstandard balancer configuration. It is moving regions during a
split.
hbase.balancer.period=6
hbase.master.balancer.stochastic.minCostNeedBalance=0.025
hbase.master.balancer.stochastic.regionCountCost=100
hbase.master.balancer.stochastic.localityCost=30
hbase.master.balancer.stochastic.moveCost=10
hbase.master.balancer.stochastic.readRequestCost=10
hbase.master.balancer.stochastic.writeRequestCost=15
I think there is a state desynchronization bug in the master when the balancer
decides to move regions around when a split is in progress.
Split procedure begins:
{noformat}
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
2021-04-30 19:11:39,874 INFO [PEWorker-65]
procedure.MasterProcedureScheduler: Took xlock for pid=79,
state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
table=IntegrationTestLoadCommonCrawl, parent=22ab65a3e71c3ab7f94c1c47067ad54e,
daughterA=2b607331c1f37f7efa21ae6c2281c89e,
daughterB=4c53f22c3fe52af4a882a5f7936aa658
{noformat}
UNASSIGN parent region (22ab65a3e71c3ab7f94c1c47067ad54e), includes WARN for
failed transition:
{noformat}
2021-04-30 19:11:39,878 INFO [PEWorker-65]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=80, ppid=79,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=22ab65a3e71c3ab7f94c1c47067ad54e,
UNASSIGN}]
2021-04-30 19:11:39,878 INFO [PEWorker-66]
procedure.MasterProcedureScheduler: Took xlock for pid=80, ppid=79,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=IntegrationTestLoadCommonCrawl, region=22ab65a3e71c3ab7f94c1c47067ad54e,
UNASSIGN
2021-04-30 19:11:39,879 INFO [PEWorker-66]
assignment.RegionStateStore: pid=80 updating hbase:meta
row=22ab65a3e71c3ab7f94c1c47067ad54e, regionState=CLOSING,
regionLocation=ip-172-31-63-65.us-west-2.compute.internal,8120,1619809716877
2021-04-30 19:11:40,032 INFO [PEWorker-66]
procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=81, ppid=80,
state=RUNNABLE; CloseRegionProcedure 22ab65a3e71c3ab7f94c1c47067ad54e,
server=ip-172-31-63-65.us-west-2.compute.internal,8120,1619809716877}]
2021-04-30 19:11:40,303 WARN
[RpcServer.priority.RWQ.Codel.write.handler=0,queue=0,port=8100]
assignment.AssignmentManager: Failed transition
org.apache.hadoop.hbase.client.DoNotRetryRegionException:
22ab65a3e71c3ab7f94c1c47067ad54e is not OPEN; state=CLOSING
at
org.apache.hadoop.hbase.master.assignment.RegionStateNode.checkOnline(RegionStateNode.java:291)
at
org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:194)
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.(SplitTableRegionProcedure.java:112)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createSplitProcedure(AssignmentManager.java:939)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionSplitTransition(AssignmentManager.java:1122)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:984)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:1019)
at
org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1727)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:16185)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
2021-04-30 19:11:44,814 INFO [PEWorker-68]
assignment.RegionStateStore: pid=80 updating hbase:meta
row=22ab65a3e71c3ab7f94c1c47067ad54e,
[GitHub] [hbase] Reidddddd commented on pull request #3212: HBASE-25831 [branch-1] remove thrift examples out of hbase-examples m…
Reidd commented on pull request #3212: URL: https://github.com/apache/hbase/pull/3212#issuecomment-830245842 > > Still related to thrift... (headache) > > Sorry :-( > But it's definitely worth doing, now that we have decided to keep thrift in 1.7.0 and also update for the CVE, while otherwise preserving Java 7 compt everywhere else. No worries, I’m tough, let’s move on! Any suggestions here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd removed a comment on pull request #3212: HBASE-25831 [branch-1] remove thrift examples out of hbase-examples m…
Reidd removed a comment on pull request #3212: URL: https://github.com/apache/hbase/pull/3212#issuecomment-830245376 No worries, I’m tough, let’s move on! Any suggestions here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on pull request #3212: HBASE-25831 [branch-1] remove thrift examples out of hbase-examples m…
Reidd commented on pull request #3212: URL: https://github.com/apache/hbase/pull/3212#issuecomment-830245376 No worries, I’m tough, let’s move on! Any suggestions here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3212: HBASE-25831 [branch-1] remove thrift examples out of hbase-examples m…
apurtell commented on pull request #3212: URL: https://github.com/apache/hbase/pull/3212#issuecomment-830237770 > Still related to thrift... (headache) Sorry :-( But it's definitely worth doing, now that we have decided to keep thrift in 1.7.0 and also update for the CVE, while otherwise preserving Java 7 compt everywhere else. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337518#comment-17337518
]
Andrew Kyle Purtell commented on HBASE-25698:
-
FWIW I was also not aware of {{hbase.blockcache.use.external}} and I think this
should be documented more prominently, because the default is counter
expectation. It totally can make sense to have L2 only the victim cache for L1,
but when operators configure offheap blockcache they could be expecting what
amounts to hbase.blockcache.use.external=true and should know about this.
> Persistent IllegalReferenceCountException at scanner open
> -
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
>
> IntegrationTestBigLinkedList
>
> COLUMN FAMILIES DESCRIPTION
>
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337516#comment-17337516
]
Andrew Kyle Purtell edited comment on HBASE-25698 at 4/30/21, 5:04 PM:
---
bq. Any chance u were using TinyLfuBlockCache?
Yes!
I have since taken that out of test config and this issue has not repeated.
bq. hbase.blockcache.use.external is not set true right
Correct, it was not set to true.
was (Author: apurtell):
bq. Any chance u were using TinyLfuBlockCache?
Yes!
I have since taken that out of test config and this issue has not repeated.
> Persistent IllegalReferenceCountException at scanner open
> -
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
>
> IntegrationTestBigLinkedList
>
> COLUMN FAMILIES DESCRIPTION
>
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337516#comment-17337516
]
Andrew Kyle Purtell commented on HBASE-25698:
-
bq. Any chance u were using TinyLfuBlockCache?
Yes!
I have since taken that out of test config and this issue has not repeated.
> Persistent IllegalReferenceCountException at scanner open
> -
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
>
> IntegrationTestBigLinkedList
>
> COLUMN FAMILIES DESCRIPTION
>
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337508#comment-17337508
]
Andrew Kyle Purtell commented on HBASE-25829:
-
The data from the test cluster is gone.
bq. Are these regions in RIT (splitting / splitting_new)?
No, there was no RIT.
bq. So far I see setRegionLocation(null) references for legit purposes like
closing region, closing it abruptly, failed open etc. Haven't seen setting this
null for splitting case.
The regions were closed. Somehow the region states persisted in the master's
data structures.
When I went to use HBCK2's {{setRegionState}} it said the state of the regions
in question was CLOSED yet each time through the balancer chore the INFO log
lines from RegionState would print.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
> {noformat}
> The HBCK chore notices them but does nothing:
> "Loaded *80 regions* from in-memory state of AssignmentManager"
> "Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
> "Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
> Yes, there are exactly 7 region state records of SPLIT state with
> server=null.
> {noformat}
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
> orphan regions
> 2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0
> orphan regions
> {noformat}
> This repeats indefinitely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337465#comment-17337465
]
Anoop Sam John commented on HBASE-25698:
Am still on it. We need to make sure all the paths are proper handling
retain/release stuff.
> Persistent IllegalReferenceCountException at scanner open
> -
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
>
> IntegrationTestBigLinkedList
>
> COLUMN FAMILIES DESCRIPTION
>
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25825) RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer
[ https://issues.apache.org/jira/browse/HBASE-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25825. --- Hadoop Flags: Reviewed Assignee: Duo Zhang Resolution: Fixed Pushed to branch-2.3+. Thanks [~niuyulin] for reviewing. > RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to > internal balancer > > > Key: HBASE-25825 > URL: https://issues.apache.org/jira/browse/HBASE-25825 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3, 2.3.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25825) RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer
[ https://issues.apache.org/jira/browse/HBASE-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25825: -- Fix Version/s: 2.3.6 2.4.3 2.5.0 3.0.0-alpha-1 > RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to > internal balancer > > > Key: HBASE-25825 > URL: https://issues.apache.org/jira/browse/HBASE-25825 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3, 2.3.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
Apache-HBase commented on pull request #3139: URL: https://github.com/apache/hbase/pull/3139#issuecomment-830158635 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 1m 25s | master passed | | +1 :green_heart: | shadedjars | 8m 55s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 57s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 34s | hbase-server generated 2 new + 18 unchanged - 2 fixed = 20 total (was 20) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 54s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 216m 51s | hbase-server in the patch passed. | | | | 253m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3139 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0054c3eccfbe 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/testReport/ | | Max. process+thread count | 2975 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
Apache-HBase commented on pull request #3139: URL: https://github.com/apache/hbase/pull/3139#issuecomment-830152140 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 44s | master passed | | +1 :green_heart: | compile | 1m 46s | master passed | | +1 :green_heart: | shadedjars | 9m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 45s | the patch passed | | +1 :green_heart: | compile | 1m 44s | the patch passed | | +1 :green_heart: | javac | 1m 44s | the patch passed | | +1 :green_heart: | shadedjars | 9m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 36s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 203m 42s | hbase-server in the patch passed. | | | | 243m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3139 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6c05678f044d 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/testReport/ | | Max. process+thread count | 3174 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25804) [branch-1] Make hbase-thrift module build with jdk8
[
https://issues.apache.org/jira/browse/HBASE-25804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337439#comment-17337439
]
Hudson commented on HBASE-25804:
Results for branch branch-1
[build #119 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/119/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/119//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/119//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/119//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> [branch-1] Make hbase-thrift module build with jdk8
> ---
>
> Key: HBASE-25804
> URL: https://issues.apache.org/jira/browse/HBASE-25804
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Blocker
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #3209: HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should cha…
Apache9 merged pull request #3209: URL: https://github.com/apache/hbase/pull/3209 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337423#comment-17337423
]
Viraj Jasani commented on HBASE-25698:
--
[~anoop.hbase] Thanks for this nice find, this sounds right to me. TinyLfu does
not perform retain() for already cached blocks unlike LruBlockCache does this
way:
{code:java}
LruCachedBlock cb = map.computeIfPresent(cacheKey, (key, val) -> {
// It will be referenced by RPC path, so increase here. NOTICE: Must do the
retain inside
// this block. because if retain outside the map#computeIfPresent, the
evictBlock may remove
// the block and release, then we're retaining a block with refCnt=0 which is
disallowed.
// see HBASE-22422.
val.getBuffer().retain();
return val;
});
{code}
Also, I am not sure about this particular testing but AFAIK I think [~apurtell]
has used TinyLfu many times in his testing. He can confirm further anyways.
{quote}hbase.blockcache.use.external is not set true right. Then only we create
CombinedBC with L2 as VictimCache for L1
{quote}
I was also not aware of this, just looked at the relevant code.
{code:java}
public InclusiveCombinedBlockCache(FirstLevelBlockCache l1, BlockCache l2) {
super(l1,l2);
l1.setVictimCache(l2);
}
{code}
I hope what you are suggesting to fix in TinyLfu is somewhat similar to this:
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
index a0dc30c524..1cb53dc6b6 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/TinyLfuBlockCache.java
@@ -170,6 +170,7 @@ public final class TinyLfuBlockCache implements
FirstLevelBlockCache {
value = victimCache.getBlock(cacheKey, caching, repeat,
updateCacheMetrics);
if ((value != null) && caching) {
if ((value instanceof HFileBlock) && ((HFileBlock)
value).isSharedMem()) {
+ value.retain();
value = HFileBlock.deepCloneOnHeap((HFileBlock) value);
}
cacheBlock(cacheKey, value);
@@ -203,6 +204,7 @@ public final class TinyLfuBlockCache implements
FirstLevelBlockCache {
@Override
public boolean evictBlock(BlockCacheKey cacheKey) {
Cacheable value = cache.asMap().remove(cacheKey);
+ value.release();
return (value != null);
}
{code}
> Persistent IllegalReferenceCountException at scanner open
> -
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
>
[GitHub] [hbase] Apache-HBase commented on pull request #3213: HBASE-25774 Added more detailed logs about the restarting of region s…
Apache-HBase commented on pull request #3213: URL: https://github.com/apache/hbase/pull/3213#issuecomment-830135677 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 57s | master passed | | +1 :green_heart: | compile | 1m 34s | master passed | | +1 :green_heart: | shadedjars | 10m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 10s | the patch passed | | +1 :green_heart: | compile | 1m 22s | the patch passed | | +1 :green_heart: | javac | 1m 22s | the patch passed | | +1 :green_heart: | shadedjars | 9m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 204m 53s | hbase-server in the patch passed. | | | | 243m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3213/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3213 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fe4dd8b94cbf 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3213/1/testReport/ | | Max. process+thread count | 3471 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3213/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3213: HBASE-25774 Added more detailed logs about the restarting of region s…
Apache-HBase commented on pull request #3213: URL: https://github.com/apache/hbase/pull/3213#issuecomment-830099776 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 9m 22s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 9m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 155m 30s | hbase-server in the patch passed. | | | | 188m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3213/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3213 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8c906241dc85 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3213/1/testReport/ | | Max. process+thread count | 4093 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3213/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337389#comment-17337389
]
Anoop Sam John commented on HBASE-25698:
[~apurtell] Any chance u were using TinyLfuBlockCache? I dont see it handle
the retain() call while getBlock(). As such we will have to fix that anyways.
> Persistent IllegalReferenceCountException at scanner open
> -
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
>
> IntegrationTestBigLinkedList
>
> COLUMN FAMILIES DESCRIPTION
>
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337372#comment-17337372
]
Anoop Sam John commented on HBASE-25698:
Also it has nothing to do with encoding as DATA blocks only gets encoded. Am
trying to rule out paths so as to narrow down and focus on only few paths
> Persistent IllegalReferenceCountException at scanner open
> -
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
>
> IntegrationTestBigLinkedList
>
> COLUMN FAMILIES DESCRIPTION
>
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25698) Persistent IllegalReferenceCountException at scanner open
[
https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337365#comment-17337365
]
Anoop Sam John commented on HBASE-25698:
So the trace is coming in Bloom check and its the release of a Bloom block
created this exception. That means no way it would have come from Off heap BC.
When offheap BC is enabled, non DATA blocks will be always in LRU on heap
cache. (hbase.blockcache.use.external is not set true right. Then only we
create CombinedBC with L2 as VictimCache for L1)
So this block under issue, would have come either from L1 LRU cache or the
block is created by a read from FS. Doing more checks in those areas. Am not
targeting BucketCache check. Pls raise concern if any.
> Persistent IllegalReferenceCountException at scanner open
> -
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to
> HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner
> open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink
> schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by:
> org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, decrement: 1
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at
> org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:249)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
>
> IntegrationTestBigLinkedList
>
> COLUMN FAMILIES DESCRIPTION
>
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE
> => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
Apache-HBase commented on pull request #3139: URL: https://github.com/apache/hbase/pull/3139#issuecomment-830043211 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 16s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 4s | master passed | | +1 :green_heart: | compile | 4m 10s | master passed | | +1 :green_heart: | checkstyle | 1m 34s | master passed | | +0 :ok: | refguide | 3m 40s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 2m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 4m 9s | the patch passed | | +1 :green_heart: | javac | 4m 9s | the patch passed | | -0 :warning: | checkstyle | 1m 10s | hbase-server: The patch generated 2 new + 11 unchanged - 2 fixed = 13 total (was 13) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 3m 42s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 19m 57s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 63m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3139 | | Optional Tests | dupname asflicense javac refguide xml spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a8d5c9ddea0d 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/artifact/yetus-general-check/output/branch-site/book.html | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3139/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3214: HBASE-23147 Branches-1 shaded artifact for mapreduce integration miss…
Apache-HBase commented on pull request #3214: URL: https://github.com/apache/hbase/pull/3214#issuecomment-830036538 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 10m 52s | Docker failed to build yetus/hbase:edccfe439a. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/3214 | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3214/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] zhe-huang opened a new pull request #3214: HBASE-23147 Branches-1 shaded artifact for mapreduce integration miss…
zhe-huang opened a new pull request #3214: URL: https://github.com/apache/hbase/pull/3214 …es MainClass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3213: HBASE-25774 Added more detailed logs about the restarting of region s…
Apache-HBase commented on pull request #3213: URL: https://github.com/apache/hbase/pull/3213#issuecomment-830025741 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 10s | master passed | | +1 :green_heart: | compile | 3m 39s | master passed | | +1 :green_heart: | checkstyle | 1m 11s | master passed | | +1 :green_heart: | spotbugs | 2m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | the patch passed | | +1 :green_heart: | compile | 3m 34s | the patch passed | | +1 :green_heart: | javac | 3m 34s | the patch passed | | +1 :green_heart: | checkstyle | 1m 5s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 47s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 32s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 52m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3213/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3213 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8d72685bc1a9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3213/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3030: HBASE-25634 The client scan frequently exceeds the quota, which cause…
Apache-HBase commented on pull request #3030: URL: https://github.com/apache/hbase/pull/3030#issuecomment-830011276 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | branch-2 passed | | +1 :green_heart: | compile | 1m 10s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 34s | branch-2 passed | | +1 :green_heart: | spotbugs | 1m 17s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | checkstyle | 0m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 14m 2s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 37m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3030/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3030 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux d4305a4dd284 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / c52c091609 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3030/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on pull request #3212: HBASE-25831 [branch-1] remove thrift examples out of hbase-examples m…
Reidd commented on pull request #3212: URL: https://github.com/apache/hbase/pull/3212#issuecomment-830009469 Still related to thrift... (headache) ``` [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase-examples:jar:1.7.0 [WARNING] 'reporting.plugins.plugin.version' for org.apache.maven.plugins:maven-javadoc-plugin is missing. @ org.apache.hbase:hbase:1.7.0, /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3212/src/pom.xml, line 2926, column 15 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] [INFO] Building Apache HBase - Examples 1.7.0 [INFO] [WARNING] The POM for org.apache.hbase:hbase-thrift:jar:1.7.0 is missing, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 5.181s [INFO] Finished at: Fri Apr 30 09:33:08 UTC 2021 [INFO] Final Memory: 41M/3926M [INFO] [WARNING] The requested profile "test-patch" could not be activated because it does not exist. [ERROR] Failed to execute goal on project hbase-examples: Could not resolve dependencies for project org.apache.hbase:hbase-examples:jar:1.7.0: Failure to find org.apache.hbase:hbase-thrift:jar:1.7.0 in https://repository.apache.org/content/repositories/releases/ was cached in the local repository, resolution will not be reattempted until the update interval of apache release has elapsed or updates are forced -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3030: HBASE-25634 The client scan frequently exceeds the quota, which cause…
Apache-HBase commented on pull request #3030: URL: https://github.com/apache/hbase/pull/3030#issuecomment-830008397 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 46s | branch-2 passed | | +1 :green_heart: | compile | 0m 34s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed | | +1 :green_heart: | javac | 0m 32s | the patch passed | | +1 :green_heart: | shadedjars | 7m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 57s | hbase-client in the patch passed. | | | | 31m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3030/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3030 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 32482b5d586c 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / c52c091609 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3030/3/testReport/ | | Max. process+thread count | 216 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3030/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3030: HBASE-25634 The client scan frequently exceeds the quota, which cause…
Apache-HBase commented on pull request #3030: URL: https://github.com/apache/hbase/pull/3030#issuecomment-830006805 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 23s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | branch-2 passed | | +1 :green_heart: | compile | 0m 27s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed | | +1 :green_heart: | javac | 0m 26s | the patch passed | | +1 :green_heart: | shadedjars | 6m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 57s | hbase-client in the patch passed. | | | | 28m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3030/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3030 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7b5e2949f750 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / c52c091609 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3030/3/testReport/ | | Max. process+thread count | 227 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3030/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 opened a new pull request #3213: HBASE-25774 Added more detailed logs about the restarting of region s…
Apache9 opened a new pull request #3213: URL: https://github.com/apache/hbase/pull/3213 …ervers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] mygood closed pull request #3020: HBASE-25634 The client frequently exceeds the quota, which causes the…
mygood closed pull request #3020: URL: https://github.com/apache/hbase/pull/3020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3212: HBASE-25831 [branch-1] remove thrift examples out of hbase-examples m…
Apache-HBase commented on pull request #3212: URL: https://github.com/apache/hbase/pull/3212#issuecomment-829984214 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 25s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 8m 14s | root in branch-1 failed. | | -1 :x: | compile | 0m 10s | hbase-examples in branch-1 failed with JDK Azul Systems, Inc.-1.8.0_262-b19. | | -1 :x: | compile | 0m 12s | hbase-examples in branch-1 failed with JDK Azul Systems, Inc.-1.7.0_272-b10. | | -1 :x: | checkstyle | 0m 12s | The patch fails to run checkstyle in hbase-examples | | -1 :x: | shadedjars | 0m 20s | branch has 7 errors when building our shaded downstream artifacts. | | -1 :x: | javadoc | 0m 9s | hbase-examples in branch-1 failed with JDK Azul Systems, Inc.-1.8.0_262-b19. | | -1 :x: | javadoc | 0m 14s | hbase-examples in branch-1 failed with JDK Azul Systems, Inc.-1.7.0_272-b10. | | +0 :ok: | spotbugs | 2m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 :x: | findbugs | 0m 13s | hbase-examples in branch-1 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 2m 8s | root in the patch failed. | | +1 :green_heart: | compile | 0m 46s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | -1 :x: | javac | 0m 37s | hbase-thrift-jdkAzulSystems,Inc.-1.7.0_272-b10 with JDK Azul Systems, Inc.-1.7.0_272-b10 generated 4 new + 100 unchanged - 2 fixed = 104 total (was 102) | | -1 :x: | checkstyle | 0m 38s | hbase-thrift: The patch generated 256 new + 0 unchanged - 0 fixed = 256 total (was 0) | | -1 :x: | whitespace | 0m 0s | The patch 1 line(s) with tabs. | | -1 :x: | xml | 0m 0s | The patch has 2 ill-formed XML file(s). | | -1 :x: | shadedjars | 0m 13s | patch has 7 errors when building our shaded downstream artifacts. | | -1 :x: | hadoopcheck | 1m 56s | The patch causes 10 errors with Hadoop v2.8.5. | | -1 :x: | hadoopcheck | 4m 35s | The patch causes 10 errors with Hadoop v2.9.2. | | +1 :green_heart: | javadoc | 0m 44s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 2m 10s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 3m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 22s | hbase-examples in the patch passed. | | -1 :x: | unit | 0m 34s | hbase-thrift in the patch failed. | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 39m 22s | | | Reason | Tests | |---:|:--| | XML | Parsing Error(s): | | | hbase-examples/pom.xml | | | hbase-thrift/pom.xml | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3212/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3212 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile spotbugs findbugs hbaseanti checkstyle | | uname | Linux 15bf27d94858 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3212/out/precommit/personality/provided.sh | | git revision | branch-1 / 0fe1c9c | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | mvninstall |
[GitHub] [hbase] ZhaoBQ commented on a change in pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ZhaoBQ commented on a change in pull request #3139:
URL: https://github.com/apache/hbase/pull/3139#discussion_r623735082
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -101,11 +106,21 @@ public void onConfigurationChange(Configuration conf) {
setConf(conf);
}
- private static int parseMinRegionCount(final Configuration conf) {
-final int parsedValue = conf.getInt(MIN_REGION_COUNT_KEY,
DEFAULT_MIN_REGION_COUNT);
+ private static int parseMergeMinRegionCount(final Configuration conf) {
+String parsedStringValue = conf.get(MERGE_MIN_REGION_COUNT_KEY);
Review comment:
And I found after conf.addDeprecation(deprecatedConf, newConf), the
conf.get(deprecatedConf) can not get value even we configured in conf file. So
if the configuration is not modified before upgrade to HBase-2.5, then the
configuration will be use default value. I don't know if I express cleared...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ZhaoBQ commented on a change in pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ZhaoBQ commented on a change in pull request #3139:
URL: https://github.com/apache/hbase/pull/3139#discussion_r623735082
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -101,11 +106,21 @@ public void onConfigurationChange(Configuration conf) {
setConf(conf);
}
- private static int parseMinRegionCount(final Configuration conf) {
-final int parsedValue = conf.getInt(MIN_REGION_COUNT_KEY,
DEFAULT_MIN_REGION_COUNT);
+ private static int parseMergeMinRegionCount(final Configuration conf) {
+String parsedStringValue = conf.get(MERGE_MIN_REGION_COUNT_KEY);
Review comment:
And I found after conf.addDeprecation(deprecatedConf, newConf), the
conf.get(deprecatedConf) can not get value even we configured in conf file. So
if the configuration is not modified before upgrade to HBase-2.5, then the
configuration will be use default value. I don't know if i express cleared...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (HBASE-25823) TestSlowLogAccessor.testHigherSlowLogs repeatable failure
[
https://issues.apache.org/jira/browse/HBASE-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani reassigned HBASE-25823:
Assignee: Viraj Jasani
> TestSlowLogAccessor.testHigherSlowLogs repeatable failure
> -
>
> Key: HBASE-25823
> URL: https://issues.apache.org/jira/browse/HBASE-25823
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 2.4.3
>
>
> {noformat}
> [ERROR] TestSlowLogAccessor.testHigherSlowLogs:211 Waiting timed out after
> [7,000] msec{noformat}
> Repeatable failure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3212: HBASE-25831 [branch-1] remove thrift examples out of hbase-examples m…
Apache-HBase commented on pull request #3212: URL: https://github.com/apache/hbase/pull/3212#issuecomment-829957335 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 5m 32s | Docker failed to build yetus/hbase:edccfe439a. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/3212 | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3212/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3202: HBASE-25803 Add compaction offload switch
Apache-HBase commented on pull request #3202: URL: https://github.com/apache/hbase/pull/3202#issuecomment-829955853 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-25714 Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | HBASE-25714 passed | | +1 :green_heart: | compile | 3m 34s | HBASE-25714 passed | | +1 :green_heart: | shadedjars | 8m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 17s | HBASE-25714 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 3m 30s | the patch passed | | +1 :green_heart: | javac | 3m 30s | the patch passed | | +1 :green_heart: | shadedjars | 8m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 49s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 19s | hbase-client in the patch passed. | | -1 :x: | unit | 216m 17s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 5m 17s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 8m 33s | hbase-shell in the patch passed. | | | | 275m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3202 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 43d5c1b8dfff 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-25714 / 4d9a360a78 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/testReport/ | | Max. process+thread count | 3087 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25831) [branch-1] remove thrift examples out of hbase-examples module for bypassing the thrift version check
[ https://issues.apache.org/jira/browse/HBASE-25831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337200#comment-17337200 ] Reid Chan commented on HBASE-25831: --- [~andrew.purt...@gmail.com] > [branch-1] remove thrift examples out of hbase-examples module for bypassing > the thrift version check > - > > Key: HBASE-25831 > URL: https://issues.apache.org/jira/browse/HBASE-25831 > Project: HBase > Issue Type: Task > Components: Thrift >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Blocker > > [ERROR] Failed to execute goal on project hbase-examples: Could not resolve > dependencies for project org.apache.hbase:hbase-examples:jar:1.7.0: Could not > find artifact org.apache.hbase:hbase-thrift:jar:1.7.0 in apache release > (https://repository.apache.org/content/repositories/releases/) -> [Help 1] > This is the msg when I tried to run make_rc.sh, we need to remove thrift > related codes from hbase-examples for making release successfully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25831) [branch-1] remove thrift examples out of hbase-examples module for bypassing the thrift version check
[ https://issues.apache.org/jira/browse/HBASE-25831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-25831: -- Description: [ERROR] Failed to execute goal on project hbase-examples: Could not resolve dependencies for project org.apache.hbase:hbase-examples:jar:1.7.0: Could not find artifact org.apache.hbase:hbase-thrift:jar:1.7.0 in apache release (https://repository.apache.org/content/repositories/releases/) -> [Help 1] This is the msg when I tried to run make_rc.sh, we need to remove thrift related codes from hbase-examples for making release successfully. > [branch-1] remove thrift examples out of hbase-examples module for bypassing > the thrift version check > - > > Key: HBASE-25831 > URL: https://issues.apache.org/jira/browse/HBASE-25831 > Project: HBase > Issue Type: Task > Components: Thrift >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Blocker > > [ERROR] Failed to execute goal on project hbase-examples: Could not resolve > dependencies for project org.apache.hbase:hbase-examples:jar:1.7.0: Could not > find artifact org.apache.hbase:hbase-thrift:jar:1.7.0 in apache release > (https://repository.apache.org/content/repositories/releases/) -> [Help 1] > This is the msg when I tried to run make_rc.sh, we need to remove thrift > related codes from hbase-examples for making release successfully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3202: HBASE-25803 Add compaction offload switch
Apache-HBase commented on pull request #3202: URL: https://github.com/apache/hbase/pull/3202#issuecomment-829953793 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-25714 Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 56s | HBASE-25714 passed | | +1 :green_heart: | compile | 4m 23s | HBASE-25714 passed | | +1 :green_heart: | shadedjars | 8m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 54s | HBASE-25714 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 44s | the patch passed | | +1 :green_heart: | compile | 4m 15s | the patch passed | | +1 :green_heart: | javac | 4m 15s | the patch passed | | +1 :green_heart: | shadedjars | 8m 26s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 1s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 2m 11s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 33s | hbase-client in the patch passed. | | -1 :x: | unit | 209m 24s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 5m 3s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 5s | hbase-shell in the patch passed. | | | | 271m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3202 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ea141dac78b7 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-25714 / 4d9a360a78 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/testReport/ | | Max. process+thread count | 3526 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd opened a new pull request #3212: HBASE-25831 [branch-1] remove thrift examples out of hbase-examples m…
Reidd opened a new pull request #3212: URL: https://github.com/apache/hbase/pull/3212 …odule for bypassing the thrift version check -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25831) [branch-1] remove thrift examples out of hbase-examples module for bypassing the thrift version check
[ https://issues.apache.org/jira/browse/HBASE-25831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-25831: -- Summary: [branch-1] remove thrift examples out of hbase-examples module for bypassing the thrift version check (was: [branch-1] remove thrift examples out of hbaes-examples module for bypassing the thrift version check) > [branch-1] remove thrift examples out of hbase-examples module for bypassing > the thrift version check > - > > Key: HBASE-25831 > URL: https://issues.apache.org/jira/browse/HBASE-25831 > Project: HBase > Issue Type: Task > Components: Thrift >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25831) [branch-1] remove thrift examples out of hbaes-examples module for bypassing the thrift version check
Reid Chan created HBASE-25831: - Summary: [branch-1] remove thrift examples out of hbaes-examples module for bypassing the thrift version check Key: HBASE-25831 URL: https://issues.apache.org/jira/browse/HBASE-25831 Project: HBase Issue Type: Task Components: Thrift Reporter: Reid Chan Assignee: Reid Chan -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829951237 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 2m 49s | master passed | | +1 :green_heart: | shadedjars | 8m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 2m 49s | the patch passed | | +1 :green_heart: | javac | 2m 49s | the patch passed | | +1 :green_heart: | shadedjars | 8m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 404m 56s | root in the patch passed. | | | | 446m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b8adc3e35a5a 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/testReport/ | | Max. process+thread count | 3951 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337191#comment-17337191
]
Viraj Jasani commented on HBASE-25829:
--
{quote}"Loaded *80 regions* from in-memory state of AssignmentManager"
"Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
{quote}
First log line comes from loading all regions from in-memory state:
loadRegionsFromInMemoryState() and second one from loadRegionsFromRSReport().
By any chance, was there any WARN log similar to _*Region is split but NOT
offline: \{regionNameAsString}*_ ?
{code:java}
private void loadRegionsFromInMemoryState() {
List regionStates =
master.getAssignmentManager().getRegionStates().getRegionStates();
for (RegionState regionState : regionStates) {
RegionInfo regionInfo = regionState.getRegion();
if (master.getTableStateManager()
.isTableState(regionInfo.getTable(), TableState.State.DISABLED)) {
disabledTableRegions.add(regionInfo.getRegionNameAsString());
}
if (regionInfo.isSplitParent()) {
splitParentRegions.add(regionInfo.getRegionNameAsString());
}
HbckRegionInfo.MetaEntry metaEntry =
new HbckRegionInfo.MetaEntry(regionInfo, regionState.getServerName(),
regionState.getStamp());
regionInfoMap.put(regionInfo.getEncodedName(), new
HbckRegionInfo(metaEntry));
}
LOG.info("Loaded {} regions from in-memory state of AssignmentManager",
regionStates.size());
}
{code}
{quote}However whenever the balancer runs there are a number of concerning INFO
level log messages printed of the form _assignment.RegionStates: Skipping, no
server for state=SPLIT, location=null, table=TABLENAME_
{quote}
Are these regions in RIT (splitting / splitting_new)?
I am trying to chase all references where we set regionLocation to null by
calling this method:
{code:java}
public ServerName setRegionLocation(final ServerName serverName) {
ServerName lastRegionLocation = this.regionLocation;
if (LOG.isTraceEnabled() && serverName == null) {
LOG.trace("Tracking when we are set to null " + this, new
Throwable("TRACE"));
}
this.regionLocation = serverName;
this.lastUpdate = EnvironmentEdgeManager.currentTime();
return lastRegionLocation;
}
{code}
So far I see setRegionLocation(null) references for legit purposes like closing
region, closing it abruptly, failed open etc. Haven't seen setting this null
for splitting case.
In the meanwhile, how do corresponding meta entries look like for these 7
regions?
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
> happened in the happy path.
> There were no errors accessing all loaded table data. The integration test
> writes a log to HDFS of every cell written to HBase and the verify phase uses
> that log to read each value and confirm it. That seems fine:
> {noformat}
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
> 2021-04-30 02:16:33,316 INFO [main]
> test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
> {noformat}
> However whenever the balancer runs there are a number of concerning INFO
> level log messages printed of the form _assignment.RegionStates: Skipping, no
> server for state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
>
[jira] [Created] (HBASE-25830) HBaseCluster support CompactionServer for UTs
Yulin Niu created HBASE-25830: - Summary: HBaseCluster support CompactionServer for UTs Key: HBASE-25830 URL: https://issues.apache.org/jira/browse/HBASE-25830 Project: HBase Issue Type: Sub-task Components: Compaction Reporter: Yulin Niu Assignee: Yulin Niu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829875446 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 53s | master passed | | +1 :green_heart: | compile | 3m 19s | master passed | | +1 :green_heart: | shadedjars | 9m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 29s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 49s | the patch passed | | +1 :green_heart: | compile | 3m 23s | the patch passed | | +1 :green_heart: | javac | 3m 23s | the patch passed | | +1 :green_heart: | shadedjars | 8m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 29s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 256m 27s | root in the patch passed. | | | | 302m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux aa27a4b10fbe 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/testReport/ | | Max. process+thread count | 4477 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ZhaoBQ commented on a change in pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ZhaoBQ commented on a change in pull request #3139:
URL: https://github.com/apache/hbase/pull/3139#discussion_r623633690
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -101,11 +106,21 @@ public void onConfigurationChange(Configuration conf) {
setConf(conf);
}
- private static int parseMinRegionCount(final Configuration conf) {
-final int parsedValue = conf.getInt(MIN_REGION_COUNT_KEY,
DEFAULT_MIN_REGION_COUNT);
+ private static int parseMergeMinRegionCount(final Configuration conf) {
+String parsedStringValue = conf.get(MERGE_MIN_REGION_COUNT_KEY);
Review comment:
The key point is how to recognize that the user has used the old
configuration and log warnings once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
