[GitHub] [hbase] Apache-HBase commented on pull request #3202: HBASE-25803 Add compaction offload switch
Apache-HBase commented on pull request #3202: URL: https://github.com/apache/hbase/pull/3202#issuecomment-829858981 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-25714 Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | HBASE-25714 passed | | +1 :green_heart: | compile | 7m 52s | HBASE-25714 passed | | +1 :green_heart: | checkstyle | 3m 5s | HBASE-25714 passed | | +1 :green_heart: | spotbugs | 9m 30s | HBASE-25714 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 7m 47s | the patch passed | | +1 :green_heart: | cc | 7m 47s | the patch passed | | +1 :green_heart: | javac | 7m 47s | the patch passed | | +1 :green_heart: | checkstyle | 0m 9s | The patch passed checkstyle in hbase-protocol-shaded | | +1 :green_heart: | checkstyle | 0m 24s | The patch passed checkstyle in hbase-common | | +1 :green_heart: | checkstyle | 0m 29s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 1m 11s | hbase-server: The patch generated 0 new + 102 unchanged - 3 fixed = 102 total (was 105) | | +1 :green_heart: | checkstyle | 0m 45s | The patch passed checkstyle in hbase-thrift | | +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-shell | | -0 :warning: | rubocop | 0m 13s | The patch generated 2 new + 377 unchanged - 0 fixed = 379 total (was 377) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 2s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 3m 20s | the patch passed | | +1 :green_heart: | spotbugs | 10m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 3s | The patch does not generate ASF License warnings. | | | | 85m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3202 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool rubocop | | uname | Linux 8906569b4674 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-25714 / 4d9a360a78 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3202/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ZhaoBQ commented on a change in pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ZhaoBQ commented on a change in pull request #3139:
URL: https://github.com/apache/hbase/pull/3139#discussion_r623577936
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -101,11 +106,21 @@ public void onConfigurationChange(Configuration conf) {
setConf(conf);
}
- private static int parseMinRegionCount(final Configuration conf) {
-final int parsedValue = conf.getInt(MIN_REGION_COUNT_KEY,
DEFAULT_MIN_REGION_COUNT);
+ private static int parseMergeMinRegionCount(final Configuration conf) {
+String parsedStringValue = conf.get(MERGE_MIN_REGION_COUNT_KEY);
Review comment:
The deprecation API is good, but has some confusion as I said in the
jira. When we call conf.addDeprecation(deprecatedConf, newConf), and only set
newConf, conf.get(deprecatedConf) always get value even it not set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ZhaoBQ commented on a change in pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ZhaoBQ commented on a change in pull request #3139: URL: https://github.com/apache/hbase/pull/3139#discussion_r623577803 ## File path: hbase-common/src/main/resources/hbase-default.xml ## @@ -638,12 +638,6 @@ possible configurations would overwhelm and obscure the important. true Whether to merge a region as part of normalization. - -hbase.normalizer.min.region.count Review comment: If here just change to "hbase.normalizer.merge.min.region.count", conf.get(MERGE_MIN_REGION_COUNT_KEY) always get value even user not configured, and we can't be backward compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25774) TestSyncReplicationStandbyKillRS#testStandbyKillRegionServer is flaky
[
https://issues.apache.org/jira/browse/HBASE-25774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337057#comment-17337057
]
Duo Zhang commented on HBASE-25774:
---
{noformat}
2021-04-28T06:22:27,806 DEBUG [Time-limited test]
replication.TestSyncReplicationStandbyKillRS(110): Going to verify the result,
1000 records expected
2021-04-28T06:22:27,808 DEBUG [RPCClient-NioEventLoopGroup-7-6]
client.AsyncRegionLocatorHelper(63): Try updating
region=hbase:meta,,1.1588230740, hostname=6839e4a4a176,39473,1619590928211,
seqNum=-1 , the old value is region=hbase:meta,,1.1588230740,
hostname=6839e4a4a176,39473,1619590928211, seqNum=-1,
error=java.net.ConnectException: Call to address=6839e4a4a176:39473null failed
on connection exception:
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 6839e4a4a176/172.17.0.3:39473
2021-04-28T06:22:27,809 DEBUG [RPCClient-NioEventLoopGroup-7-6]
client.AsyncRegionLocatorHelper(71): The actual exception when updating
region=hbase:meta,,1.1588230740, hostname=6839e4a4a176,39473,1619590928211,
seqNum=-1 is java.net.ConnectException: Connection refused
2021-04-28T06:22:27,809 DEBUG [RPCClient-NioEventLoopGroup-7-6]
client.AsyncRegionLocatorHelper(87): Try removing
region=hbase:meta,,1.1588230740, hostname=6839e4a4a176,39473,1619590928211,
seqNum=-1 from cache
2021-04-28T06:22:27,809 DEBUG [RPCClient-NioEventLoopGroup-7-6]
ipc.FailedServers(53): Added failed server with address 6839e4a4a176:39473 to
list caused by
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: 6839e4a4a176/172.17.0.3:39473
2021-04-28T06:22:27,823 DEBUG
[RS:3;6839e4a4a176:42643.replicationSource.wal-reader.6839e4a4a176%2C42643%2C1619590943524,1]
wal.ProtobufLogReader(422): EOF at position 381
2021-04-28T06:22:27,848 DEBUG
[RS:4;6839e4a4a176:44305.replicationSource.wal-reader.6839e4a4a176%2C44305%2C1619590944923,1]
wal.ProtobufLogReader(422): EOF at position 83
2021-04-28T06:22:27,894 DEBUG
[RS:5;6839e4a4a176:45961.replicationSource.wal-reader.6839e4a4a176%2C45961%2C1619590947375,1]
wal.ProtobufLogReader(422): EOF at position 83
2021-04-28T06:22:27,916 DEBUG [Async-Client-Retry-Timer-pool-0]
client.ConnectionUtils(548): Start fetching meta region location from registry
2021-04-28T06:22:27,920 DEBUG [RPCClient-NioEventLoopGroup-7-4]
client.ConnectionUtils(556): The fetched meta region location is
[region=hbase:meta,,1.1588230740, hostname=6839e4a4a176,44305,1619590944923,
seqNum=-1]
2021-04-28T06:22:27,921 DEBUG [RPCClient-NioEventLoopGroup-7-4]
ipc.RpcConnection(122): Using SIMPLE authentication for service=ClientService,
sasl=false
2021-04-28T06:22:27,924 INFO [RS-EventLoopGroup-14-2]
ipc.ServerRpcConnection(557): Connection from 172.17.0.3:37390,
version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE),
service=ClientService
2021-04-28T06:22:27,932 DEBUG
[RS:3;6839e4a4a176:42643.replicationSource.wal-reader.6839e4a4a176%2C42643%2C1619590943524,1]
wal.ProtobufLogReader(422): EOF at position 381
2021-04-28T06:22:27,955 DEBUG
[RS:4;6839e4a4a176:44305.replicationSource.wal-reader.6839e4a4a176%2C44305%2C1619590944923,1]
wal.ProtobufLogReader(422): EOF at position 83
2021-04-28T06:22:27,965 DEBUG [RPCClient-NioEventLoopGroup-7-10]
client.AsyncNonMetaRegionLocator(355): The fetched location of 'SyncRep',
row='\x00\x00\x00\x00', locateType=CURRENT is
[region=SyncRep,,1619590931098.3c8d57984856848ef8526f4bc5b3fbb8.,
hostname=6839e4a4a176,42643,1619590943524, seqNum=9]
2021-04-28T06:22:27,967 DEBUG [RPCClient-NioEventLoopGroup-7-10]
ipc.RpcConnection(122): Using SIMPLE authentication for service=ClientService,
sasl=false
2021-04-28T06:22:27,970 INFO [RS-EventLoopGroup-13-1]
ipc.ServerRpcConnection(557): Connection from 172.17.0.3:58964,
version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE),
service=ClientService
2021-04-28T06:22:27,971 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42643] ipc.CallRunner(149):
callId: 28 service: ClientService methodName: Get size: 107 connection:
172.17.0.3:58964 deadline: 1619591007970,
exception=org.apache.hadoop.hbase.DoNotRetryIOException:
SyncRep,,1619590931098.3c8d57984856848ef8526f4bc5b3fbb8. is in STANDBY state.
2021-04-28T06:22:28,018 DEBUG
[RS:5;6839e4a4a176:45961.replicationSource.wal-reader.6839e4a4a176%2C45961%2C1619590947375,1]
wal.ProtobufLogReader(422): EOF at position 83
2021-04-28T06:22:28,072 DEBUG
[RS:3;6839e4a4a176:42643.replicationSource.wal-reader.6839e4a4a176%2C42643%2C1619590943524,1]
wal.ProtobufLogReader(422): EOF at position 381
2021-04-28T06:22:28,072 DEBUG
[RS:4;6839e4a4a176:44305.replicationSource.wal-reader.6839e4a4a176%2C44305%2C1619590944923,1]
wal.ProtobufLogReader(422): EOF at position 83
{noformat}
Seems the problem is that, we failed in the verify stage, and then we scheduled
another
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829759831 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 40s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 42s | master passed | | +1 :green_heart: | compile | 8m 30s | master passed | | +1 :green_heart: | checkstyle | 1m 54s | master passed | | +1 :green_heart: | spotbugs | 10m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 52s | the patch passed | | +1 :green_heart: | compile | 8m 17s | the patch passed | | -0 :warning: | javac | 8m 17s | root generated 20 new + 1541 unchanged - 20 fixed = 1561 total (was 1561) | | -0 :warning: | checkstyle | 1m 56s | root: The patch generated 20 new + 0 unchanged - 0 fixed = 20 total (was 0) | | -0 :warning: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 19m 30s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 11m 32s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 80m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux f43a0cd05f2b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 141 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25829:
Description:
Seen after an integration test (see HBASE-25824) with 'calm' monkey, so this
happened in the happy path.
There were no errors accessing all loaded table data. The integration test
writes a log to HDFS of every cell written to HBase and the verify phase uses
that log to read each value and confirm it. That seems fine:
{noformat}
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
{noformat}
However whenever the balancer runs there are a number of concerning INFO level
log messages printed of the form _assignment.RegionStates: Skipping, no server
for state=SPLIT, location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
{noformat}
The HBCK chore notices them but does nothing:
"Loaded *80 regions* from in-memory state of AssignmentManager"
"Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
"Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
Yes, there are exactly 7 region state records of SPLIT state with server=null.
{noformat}
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
orphan regions
2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0 orphan
regions
{noformat}
This repeats indefinitely.
was:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data. The integration test
writes a log to HDFS of every cell written to HBase and the verify phase uses
that log to read each value and confirm it. That seems fine:
{noformat}
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
{noformat}
However whenever the balancer runs there are a number of concerning INFO level
log messages printed of the form _assignment.RegionStates: Skipping, no server
for state=SPLIT, location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for
[jira] [Updated] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25829:
Description:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data. The integration test
writes a log to HDFS of every cell written to HBase and the verify phase uses
that log to read each value and confirm it. That seems fine:
{noformat}
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: REFERENCED: 154943544
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: UNREFERENCED: 0
2021-04-30 02:16:33,316 INFO [main]
test.IntegrationTestLoadCommonCrawl$Verify: CORRUPT: 0
{noformat}
However whenever the balancer runs there are a number of concerning INFO level
log messages printed of the form _assignment.RegionStates: Skipping, no server
for state=SPLIT, location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
{noformat}
The HBCK chore notices them but does nothing:
"Loaded *80 regions* from in-memory state of AssignmentManager"
"Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
"Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
Yes, there are exactly 7 region state records of SPLIT state with server=null.
{noformat}
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
orphan regions
2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0 orphan
regions
{noformat}
This repeats indefinitely.
was:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data, however whenever the
balancer runs there are a number of concerning INFO level log messages printed
of the form _assignment.RegionStates: Skipping, no server for state=SPLIT,
location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
[jira] [Comment Edited] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337022#comment-17337022
]
Andrew Kyle Purtell edited comment on HBASE-25829 at 4/30/21, 2:15 AM:
---
Also, I didn't see a HBCK2 option that does anything for this case.
{{./bin/hbase hbck2 -j /path/to/jar fixMeta}} printed a message about kicking
the chore. The chore didn't handle it.
{{setRegionState}} doesn't seem right. Maybe a {{removeRegionState}} if we had
the option.
was (Author: apurtell):
Also, I didn't see a HBCK2 option that does anything for this case.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test load with 'calm' monkey, so this happened in
> the happy path.
> There were no errors accessing all loaded table data, however whenever the
> balancer runs there are a number of concerning INFO level log messages
> printed of the form _assignment.RegionStates: Skipping, no server for
> state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
> {noformat}
> The HBCK chore notices them but does nothing:
> "Loaded *80 regions* from in-memory state of AssignmentManager"
> "Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
> "Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
> Yes, there are exactly 7 region state records of SPLIT state with
> server=null.
> {noformat}
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
> orphan regions
> 2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0
> orphan regions
> {noformat}
> This repeats indefinitely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337022#comment-17337022
]
Andrew Kyle Purtell commented on HBASE-25829:
-
Also, I didn't see a HBCK2 option that does anything for this case.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test load with 'calm' monkey, so this happened in
> the happy path.
> There were no errors accessing all loaded table data, however whenever the
> balancer runs there are a number of concerning INFO level log messages
> printed of the form _assignment.RegionStates: Skipping, no server for
> state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
> {noformat}
> The HBCK chore notices them but does nothing:
> "Loaded *80 regions* from in-memory state of AssignmentManager"
> "Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
> "Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
> Yes, there are exactly 7 region state records of SPLIT state with
> server=null.
> {noformat}
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
> orphan regions
> 2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0
> orphan regions
> {noformat}
> This repeats indefinitely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337021#comment-17337021
]
Andrew Kyle Purtell commented on HBASE-25829:
-
[~vjasani] [~stack] This might be of interest to you.
I'll start looking at the code tomorrow.
> SPLIT state detritus
>
>
> Key: HBASE-25829
> URL: https://issues.apache.org/jira/browse/HBASE-25829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Seen after an integration test load with 'calm' monkey, so this happened in
> the happy path.
> There were no errors accessing all loaded table data, however whenever the
> balancer runs there are a number of concerning INFO level log messages
> printed of the form _assignment.RegionStates: Skipping, no server for
> state=SPLIT, location=null, table=TABLENAME_
> For example:
> {noformat}
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
> 2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
> assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
> table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
> {noformat}
> The HBCK chore notices them but does nothing:
> "Loaded *80 regions* from in-memory state of AssignmentManager"
> "Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
> "Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
> Yes, there are exactly 7 region state records of SPLIT state with
> server=null.
> {noformat}
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
> 2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
> orphan regions
> 2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
> master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0
> orphan regions
> {noformat}
> This repeats indefinitely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25829:
Description:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data, however whenever the
balancer runs there are a number of concerning INFO level log messages printed
of the form _assignment.RegionStates: Skipping, no server for state=SPLIT,
location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
{noformat}
The HBCK chore notices them but does nothing:
"Loaded *80 regions* from in-memory state of AssignmentManager"
"Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
"Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
Yes, there are exactly 7 region state records of SPLIT state with server=null.
{noformat}
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
orphan regions
2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0 orphan
regions
{noformat}
This repeats indefinitely.
was:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data, however whenever the
balancer runs there are a number of concerning INFO level log messages printed
of the form _assignment.RegionStates: Skipping, no server for state=SPLIT,
location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl,
[jira] [Updated] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25829:
Description:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data, however whenever the
balancer runs there are a number of concerning INFO level log messages printed
of the form _assignment.RegionStates: Skipping, no server for state=SPLIT,
location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
{noformat}
The HBCK chore notices them but does nothing:
"Loaded *80 regions* from in-memory state of AssignmentManager"
"Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
"Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
Yes, there are exactly 7 region state records of SPLIT state with server=null.
{noformat}
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
orphan regions
2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0 orphan
regions
{noformat}
This repeats indefinitely.
was:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data, however whenever the
balancer runs there are a number of concerning INFO level log messages printed
of the form _assignment.RegionStates: Skipping, no server for state=SPLIT,
location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl,
[jira] [Created] (HBASE-25829) SPLIT state detritus
Andrew Kyle Purtell created HBASE-25829:
---
Summary: SPLIT state detritus
Key: HBASE-25829
URL: https://issues.apache.org/jira/browse/HBASE-25829
Project: HBase
Issue Type: Bug
Affects Versions: 2.4.3
Reporter: Andrew Kyle Purtell
Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data, however whenever the
balancer runs there are a number of concerning INFO level log messages printed
of the form _assignment.RegionStates: Skipping, no server for state=SPLIT,
location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
{noformat}
The HBCK chore notices them but does nothing:
"Loaded *80 regions *from in-memory state of AssignmentManager"
"Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
"Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
{noformat}
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
orphan regions
2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0 orphan
regions
{noformat}
This repeats indefinitely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25829) SPLIT state detritus
[
https://issues.apache.org/jira/browse/HBASE-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25829:
Description:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data, however whenever the
balancer runs there are a number of concerning INFO level log messages printed
of the form _assignment.RegionStates: Skipping, no server for state=SPLIT,
location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
{noformat}
The HBCK chore notices them but does nothing:
"Loaded *80 regions* from in-memory state of AssignmentManager"
"Loaded *73 regions from 5 regionservers' reports* and found 0 orphan regions"
"Loaded 3 tables 80 regions from filesystem and found 0 orphan regions"
{noformat}
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 80 regions from in-memory state of AssignmentManager
2021-04-30 02:02:09,300 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 73 regions from 5 regionservers' reports and found 0
orphan regions
2021-04-30 02:02:09,306 INFO [master/ip-172-31-58-47:8100.Chore.1]
master.HbckChore: Loaded 3 tables 80 regions from filesystem and found 0 orphan
regions
{noformat}
This repeats indefinitely.
was:
Seen after an integration test load with 'calm' monkey, so this happened in the
happy path.
There were no errors accessing all loaded table data, however whenever the
balancer runs there are a number of concerning INFO level log messages printed
of the form _assignment.RegionStates: Skipping, no server for state=SPLIT,
location=null, table=TABLENAME_
For example:
{noformat}
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=087fb2f7847c2fc0a0b85eb30a97036e
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=0952b94a920454afe9c40becbb7bf205
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=f87a8b993f7eca2524bf2331b7ee3c06
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=74bb28864a120decdf0f4956741df745
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=bc918b609ade0ae4d5530f0467354cae
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=183a199984539f3917a2f8927fe01572
2021-04-30 02:02:09,286 INFO [master/ip-172-31-58-47:8100.Chore.2]
assignment.RegionStates: Skipping, no server for state=SPLIT, location=null,
table=IntegrationTestLoadCommonCrawl, region=6cc5ce4fb4adc00445b3ec7dd8760ba8
{noformat}
The HBCK chore notices them but
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829735076 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 37s | master passed | | +1 :green_heart: | compile | 3m 1s | master passed | | +1 :green_heart: | shadedjars | 8m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 40s | the patch passed | | +1 :green_heart: | compile | 3m 11s | the patch passed | | +1 :green_heart: | javac | 3m 11s | the patch passed | | +1 :green_heart: | shadedjars | 10m 27s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 57s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 293m 21s | root in the patch failed. | | | | 339m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f35481672a04 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/testReport/ | | Max. process+thread count | 3937 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25824) IntegrationTestLoadCommonCrawl
[
https://issues.apache.org/jira/browse/HBASE-25824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25824:
Description:
This integration test loads successful resource retrieval records from the
Common Crawl (https://commoncrawl.org/) public dataset into an HBase table and
writes records that can be used to later verify the presence and integrity of
those records.
Run like:
{noformat}
./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl \
-files /path/to/hadoop-aws.jar \
-Dfs.s3n.awsAccessKeyId= \
-Dfs.s3n.awsSecretAccessKey= \
/path/to/test-CC-MAIN-2021-10-warc.paths.gz \
/path/to/tmp/warc-loader-output
{noformat}
Access to the Common Crawl dataset in S3 is made available to anyone by Amazon
AWS, but Hadoop's S3N filesystem still requires valid access credentials to
initialize.
The input path can either specify a directory or a file. The file may
optionally be compressed with gzip. If a directory, the loader expects the
directory to contain one or more WARC files from the Common Crawl dataset. If a
file, the loader expects a list of Hadoop S3N URIs which point to S3 locations
for one or more WARC files from the Common Crawl dataset, one URI per line.
Lines should be terminated with the UNIX line terminator.
Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz is a list
of all WARC files comprising the Q1 2021 crawl archive. There are 64,000 WARC
files in this data set, each containing ~1GB of gzipped data. The WARC files
contain several record types, such as metadata, request, and response, but we
only load the response record types. If the HBase table schema does not specify
compression (by default) there is roughly a 10x expansion. Loading the complete
CC-MAIN-2021-10 crawl archive results in a table approximately 640 TB in size.
The hadoop-aws jar will be needed at runtime to instantiate the S3N filesystem.
Use the -files ToolRunner argument to add it.
You can also split the Loader and Verify stages:
Load with:
{noformat}
./bin/hbase
'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Loader' \
-files /path/to/hadoop-aws.jar \
-Dfs.s3n.awsAccessKeyId= \
-Dfs.s3n.awsSecretAccessKey= \
/path/to/test-CC-MAIN-2021-10-warc.paths.gz \
/path/to/tmp/warc-loader-output
{noformat}
Verify with:
{noformat}
./bin/hbase
'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Verify' \
/path/to/tmp/warc-loader-output
{noformat}
was:
This integration test loads successful resource retrieval records from the
Common Crawl (https://commoncrawl.org/) public dataset into an HBase table and
writes records that can be used to later verify the presence and integrity of
those records.
Run like:
{noformat}
./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl \
-Dfs.s3n.awsAccessKeyId= \
-Dfs.s3n.awsSecretAccessKey= \
/path/to/test-CC-MAIN-2021-10-warc.paths.gz \
/path/to/tmp/warc-loader-output
{noformat}
Access to the Common Crawl dataset in S3 is made available to anyone by Amazon
AWS, but Hadoop's S3N filesystem still requires valid access credentials to
initialize.
The input path can either specify a directory or a file. The file may
optionally be compressed with gzip. If a directory, the loader expects the
directory to contain one or more WARC files from the Common Crawl dataset. If a
file, the loader expects a list of Hadoop S3N URIs which point to S3 locations
for one or more WARC files from the Common Crawl dataset, one URI per line.
Lines should be terminated with the UNIX line terminator.
Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz is a list
of all WARC files comprising the Q1 2021 crawl archive. There are 64,000 WARC
files in this data set, each containing ~1GB of gzipped data. The WARC files
contain several record types, such as metadata, request, and response, but we
only load the response record types. If the HBase table schema does not specify
compression (by default) there is roughly a 10x expansion. Loading the full
crawl archive results in a table approximately 640 TB in size.
The hadoop-aws jar will be needed at runtime to instantiate the S3N filesystem.
Use the -files ToolRunner argument to add it.
You can also split the Loader and Verify stages:
Load with:
{noformat}
./bin/hbase
'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Loader' \
-files /path/to/hadoop-aws.jar \
-Dfs.s3n.awsAccessKeyId= \
-Dfs.s3n.awsSecretAccessKey= \
/path/to/test-CC-MAIN-2021-10-warc.paths.gz \
/path/to/tmp/warc-loader-output
{noformat}
Verify with:
{noformat}
./bin/hbase
'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Verify' \
/path/to/tmp/warc-loader-output
{noformat}
> IntegrationTestLoadCommonCrawl
> --
>
> Key: HBASE-25824
> URL:
[jira] [Commented] (HBASE-25828) CompactionProgress WARNS: "totalCompactingKVs=N less than currentCompactedKVs=M"
[
https://issues.apache.org/jira/browse/HBASE-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337001#comment-17337001
]
Andrew Kyle Purtell commented on HBASE-25828:
-
/cc [~ndimiduk] similar to HBASE-25642, I think
> CompactionProgress WARNS: "totalCompactingKVs=N less than
> currentCompactedKVs=M"
>
>
> Key: HBASE-25828
> URL: https://issues.apache.org/jira/browse/HBASE-25828
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> Similar to HBASE-25642, but compaction progress warnings.
> Lots of warnings like:
> {noformat}
> 2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=15589 less than
> currentCompactedKVs=21411
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=21916 less than
> currentCompactedKVs=33328
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=89731 less than
> currentCompactedKVs=92808
> {noformat}
> A couple of issues:
> - Is there a way to make CompactionProgress more reliable? I seem to recall
> this is the second or third time around the block on this.
> - This is annoying because compaction progress isn't always accurate, but
> this information is not used to determine anything significant, so WARN level
> is a bit much. (Or, if this is a real correctness problem, see point above.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25642) Fix or stop warning about already cached block
[
https://issues.apache.org/jira/browse/HBASE-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25642:
Component/s: Operability
> Fix or stop warning about already cached block
> --
>
> Key: HBASE-25642
> URL: https://issues.apache.org/jira/browse/HBASE-25642
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Operability, regionserver
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> Our logs have as a fairly common occurrence:
> {noformat}
> 2021-03-05 22:24:31,034 WARN [StoreFileOpener-foo-1] hfile.BlockCacheUtil:
> Caching an already cached block: blah.bub. This is harmless and can happen in
> rare cases (see HBASE-8547)
> {noformat}
> If this is really harmless, why do we log? If it's not actually harmless,
> let's take another pass at fixing this situation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25828) CompactionProgress WARNS: "totalCompactingKVs=N less than currentCompactedKVs=M"
[
https://issues.apache.org/jira/browse/HBASE-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25828:
Description:
Similar to HBASE-25642, but compaction progress warnings.
Lots of warnings like:
{noformat}
2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=15589 less than
currentCompactedKVs=21411
{noformat}
{noformat}
2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=21916 less than
currentCompactedKVs=33328
{noformat}
{noformat}
2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=89731 less than
currentCompactedKVs=92808
{noformat}
A couple of issues:
- Is there a way to make CompactionProgress more reliable? I seem to recall
this is the second or third time around the block on this.
- This is annoying because compaction progress isn't always accurate, but this
information is not used to determine anything significant, so WARN level is a
bit much. (Or, if this is a real correctness problem, see point above.)
was:
Lots of warnings like:
{noformat}
2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=15589 less than
currentCompactedKVs=21411
{noformat}
{noformat}
2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=21916 less than
currentCompactedKVs=33328
{noformat}
{noformat}
2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=89731 less than
currentCompactedKVs=92808
{noformat}
A couple of issues:
- Is there a way to make CompactionProgress more reliable? I seem to recall
this is the second or third time around the block on this.
- This is annoying because compaction progress isn't always accurate, but this
information is not used to determine anything significant, so WARN level is a
bit much. (Or, if this is a real correctness problem, see point above.)
> CompactionProgress WARNS: "totalCompactingKVs=N less than
> currentCompactedKVs=M"
>
>
> Key: HBASE-25828
> URL: https://issues.apache.org/jira/browse/HBASE-25828
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> Similar to HBASE-25642, but compaction progress warnings.
> Lots of warnings like:
> {noformat}
> 2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=15589 less than
> currentCompactedKVs=21411
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=21916 less than
> currentCompactedKVs=33328
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=89731 less than
> currentCompactedKVs=92808
> {noformat}
> A couple of issues:
> - Is there a way to make CompactionProgress more reliable? I seem to recall
> this is the second or third time around the block on this.
> - This is annoying because compaction progress isn't always accurate, but
> this information is not used to determine anything significant, so WARN level
> is a bit much. (Or, if this is a real correctness problem, see point above.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25828) CompactionProgress WARNS: "totalCompactingKVs=N less than currentCompactedKVs=M"
Andrew Kyle Purtell created HBASE-25828:
---
Summary: CompactionProgress WARNS: "totalCompactingKVs=N less than
currentCompactedKVs=M"
Key: HBASE-25828
URL: https://issues.apache.org/jira/browse/HBASE-25828
Project: HBase
Issue Type: Bug
Affects Versions: 2.4.3
Reporter: Andrew Kyle Purtell
Fix For: 2.4.4, 3.0.0-alpha-1, 2.5.0
Lots of warnings like:
{noformat}
2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=15589 less than
currentCompactedKVs=21411
{noformat}
{noformat}
2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=21916 less than
currentCompactedKVs=33328
{noformat}
{noformat}
2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
compactions.CompactionProgress: totalCompactingKVs=89731 less than
currentCompactedKVs=92808
{noformat}
A couple of issues:
- Is there a way to make CompactionProgress more reliable? I seem to recall
this is the second or third time around the block on this.
- This is annoying because compaction progress isn't always accurate, but this
information is not used to determine anything significant, so WARN level is a
bit much. (Or, if this is a real correctness problem, see point above.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25642) Fix or stop warning about already cached block
[
https://issues.apache.org/jira/browse/HBASE-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25642:
Fix Version/s: 2.4.4
2.5.0
3.0.0-alpha-1
> Fix or stop warning about already cached block
> --
>
> Key: HBASE-25642
> URL: https://issues.apache.org/jira/browse/HBASE-25642
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, regionserver
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> Our logs have as a fairly common occurrence:
> {noformat}
> 2021-03-05 22:24:31,034 WARN [StoreFileOpener-foo-1] hfile.BlockCacheUtil:
> Caching an already cached block: blah.bub. This is harmless and can happen in
> rare cases (see HBASE-8547)
> {noformat}
> If this is really harmless, why do we log? If it's not actually harmless,
> let's take another pass at fixing this situation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25828) CompactionProgress WARNS: "totalCompactingKVs=N less than currentCompactedKVs=M"
[
https://issues.apache.org/jira/browse/HBASE-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25828:
Component/s: regionserver
Compaction
> CompactionProgress WARNS: "totalCompactingKVs=N less than
> currentCompactedKVs=M"
>
>
> Key: HBASE-25828
> URL: https://issues.apache.org/jira/browse/HBASE-25828
> Project: HBase
> Issue Type: Bug
> Components: Compaction, regionserver
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> Similar to HBASE-25642, but compaction progress warnings.
> Lots of warnings like:
> {noformat}
> 2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=15589 less than
> currentCompactedKVs=21411
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=21916 less than
> currentCompactedKVs=33328
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=89731 less than
> currentCompactedKVs=92808
> {noformat}
> A couple of issues:
> - Is there a way to make CompactionProgress more reliable? I seem to recall
> this is the second or third time around the block on this.
> - This is annoying because compaction progress isn't always accurate, but
> this information is not used to determine anything significant, so WARN level
> is a bit much. (Or, if this is a real correctness problem, see point above.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829701832 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 42s | master passed | | +1 :green_heart: | compile | 3m 58s | master passed | | +1 :green_heart: | shadedjars | 11m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 24s | the patch passed | | +1 :green_heart: | compile | 3m 54s | the patch passed | | +1 :green_heart: | javac | 3m 54s | the patch passed | | +1 :green_heart: | shadedjars | 10m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 176m 33s | root in the patch passed. | | | | 231m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9214765b4568 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/testReport/ | | Max. process+thread count | 6585 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25828) CompactionProgress WARNS: "totalCompactingKVs=N less than currentCompactedKVs=M"
[
https://issues.apache.org/jira/browse/HBASE-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25828:
Component/s: Operability
> CompactionProgress WARNS: "totalCompactingKVs=N less than
> currentCompactedKVs=M"
>
>
> Key: HBASE-25828
> URL: https://issues.apache.org/jira/browse/HBASE-25828
> Project: HBase
> Issue Type: Bug
> Components: Compaction, Operability, regionserver
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> Similar to HBASE-25642, but compaction progress warnings.
> Lots of warnings like:
> {noformat}
> 2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=15589 less than
> currentCompactedKVs=21411
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=21916 less than
> currentCompactedKVs=33328
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=89731 less than
> currentCompactedKVs=92808
> {noformat}
> A couple of issues:
> - Is there a way to make CompactionProgress more reliable? I seem to recall
> this is the second or third time around the block on this.
> - This is annoying because compaction progress isn't always accurate, but
> this information is not used to determine anything significant, so WARN level
> is a bit much. (Or, if this is a real correctness problem, see point above.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] apurtell edited a comment on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
apurtell edited a comment on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829709051 Thanks for the review @saintstack . Last precommit looked good even though earlier test failures weren't related. I made a push of improvements. I see a round of checkstyle and whitespace fixes are due. Let me address them and then close this out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
apurtell edited a comment on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829709051 Thanks for the review @saintstack . Last precommit looked good even though earlier test failures weren't related. I made a push of miscellaneous improvements. I see a round of checkstyle and whitespace fixes are due. Let me address them and then close this out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
apurtell commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829709051 Thanks for the review @saintstack . Lasts precommit looked good even though earlier test failures weren't related. I made a push of miscellaneous improvements. I see a round of checkstyle and whitespace fixes are due. Let me address them and then close this out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Kota-SH commented on pull request #3211: HBASE-25816: Improve the documentation of Architecture section of reference guide
Kota-SH commented on pull request #3211: URL: https://github.com/apache/hbase/pull/3211#issuecomment-829641449 Hi @jatsakthi @petersomogyi, Could you please review this PR and provide feedback? Thanks, Shanmukha K -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3210: HBASE-25827 Per Cell TTL tags get duplicated with increments causing tags length overflow
Apache-HBase commented on pull request #3210: URL: https://github.com/apache/hbase/pull/3210#issuecomment-829632739 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 57s | master passed | | +1 :green_heart: | compile | 1m 8s | master passed | | +1 :green_heart: | shadedjars | 9m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 9m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 223m 35s | hbase-server in the patch failed. | | | | 260m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3210 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dfbaa6ad4ed9 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/testReport/ | | Max. process+thread count | 3342 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25816) Improve the documentation of "Architecture" section of reference guide
[ https://issues.apache.org/jira/browse/HBASE-25816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335833#comment-17335833 ] Shanmukha Haripriya Kota commented on HBASE-25816: -- I have created a [PR|https://github.com/apache/hbase/pull/3211] for this. Please review. Thanks, Shanmukha K > Improve the documentation of "Architecture" section of reference guide > -- > > Key: HBASE-25816 > URL: https://issues.apache.org/jira/browse/HBASE-25816 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Shanmukha Haripriya Kota >Assignee: Shanmukha Haripriya Kota >Priority: Minor > > While going through the reference guide, I found these small changes that can > be made to the documentation of the "Architecture" part of it. Hence I'm > filing this Jira to track those changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25816) Improve the documentation of "Architecture" section of reference guide
[ https://issues.apache.org/jira/browse/HBASE-25816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shanmukha Haripriya Kota updated HBASE-25816: - Status: Patch Available (was: Open) > Improve the documentation of "Architecture" section of reference guide > -- > > Key: HBASE-25816 > URL: https://issues.apache.org/jira/browse/HBASE-25816 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Shanmukha Haripriya Kota >Assignee: Shanmukha Haripriya Kota >Priority: Minor > > While going through the reference guide, I found these small changes that can > be made to the documentation of the "Architecture" part of it. Hence I'm > filing this Jira to track those changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25816) Improve the documentation of "Architecture" section of reference guide
[ https://issues.apache.org/jira/browse/HBASE-25816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shanmukha Haripriya Kota updated HBASE-25816: - Status: Patch Available (was: Open) > Improve the documentation of "Architecture" section of reference guide > -- > > Key: HBASE-25816 > URL: https://issues.apache.org/jira/browse/HBASE-25816 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Shanmukha Haripriya Kota >Assignee: Shanmukha Haripriya Kota >Priority: Minor > > While going through the reference guide, I found these small changes that can > be made to the documentation of the "Architecture" part of it. Hence I'm > filing this Jira to track those changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25816) Improve the documentation of "Architecture" section of reference guide
[ https://issues.apache.org/jira/browse/HBASE-25816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shanmukha Haripriya Kota updated HBASE-25816: - Status: Open (was: Patch Available) > Improve the documentation of "Architecture" section of reference guide > -- > > Key: HBASE-25816 > URL: https://issues.apache.org/jira/browse/HBASE-25816 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Shanmukha Haripriya Kota >Assignee: Shanmukha Haripriya Kota >Priority: Minor > > While going through the reference guide, I found these small changes that can > be made to the documentation of the "Architecture" part of it. Hence I'm > filing this Jira to track those changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work stopped] (HBASE-25816) Improve the documentation of "Architecture" section of reference guide
[ https://issues.apache.org/jira/browse/HBASE-25816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25816 stopped by Shanmukha Haripriya Kota. > Improve the documentation of "Architecture" section of reference guide > -- > > Key: HBASE-25816 > URL: https://issues.apache.org/jira/browse/HBASE-25816 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Shanmukha Haripriya Kota >Assignee: Shanmukha Haripriya Kota >Priority: Minor > > While going through the reference guide, I found these small changes that can > be made to the documentation of the "Architecture" part of it. Hence I'm > filing this Jira to track those changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-25816) Improve the documentation of "Architecture" section of reference guide
[ https://issues.apache.org/jira/browse/HBASE-25816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25816 started by Shanmukha Haripriya Kota. > Improve the documentation of "Architecture" section of reference guide > -- > > Key: HBASE-25816 > URL: https://issues.apache.org/jira/browse/HBASE-25816 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Shanmukha Haripriya Kota >Assignee: Shanmukha Haripriya Kota >Priority: Minor > > While going through the reference guide, I found these small changes that can > be made to the documentation of the "Architecture" part of it. Hence I'm > filing this Jira to track those changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829597359 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 53s | master passed | | +1 :green_heart: | compile | 8m 21s | master passed | | +1 :green_heart: | checkstyle | 1m 56s | master passed | | +1 :green_heart: | spotbugs | 9m 44s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 8m 17s | the patch passed | | -0 :warning: | javac | 8m 17s | root generated 20 new + 1541 unchanged - 20 fixed = 1561 total (was 1561) | | -0 :warning: | checkstyle | 2m 3s | root: The patch generated 376 new + 0 unchanged - 0 fixed = 376 total (was 0) | | -0 :warning: | whitespace | 0m 1s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -0 :warning: | whitespace | 0m 1s | The patch 1 line(s) with tabs. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 18m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 10m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 77m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux e8cf1d66e54b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/artifact/yetus-general-check/output/whitespace-eol.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/artifact/yetus-general-check/output/whitespace-tabs.txt | | Max. process+thread count | 141 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3211: HBASE-25816: Improve the documentation of Architecture section of reference guide
Apache-HBase commented on pull request #3211: URL: https://github.com/apache/hbase/pull/3211#issuecomment-829593249 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +0 :ok: | refguide | 3m 28s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 25s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 18s | The patch does not generate ASF License warnings. | | | | 17m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3211/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3211 | | JIRA Issue | HBASE-25816 | | Optional Tests | dupname asflicense refguide | | uname | Linux 4ab5eb7a0d96 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3211/1/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3211/1/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3211/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3210: HBASE-25827 Per Cell TTL tags get duplicated with increments causing tags length overflow
Apache-HBase commented on pull request #3210: URL: https://github.com/apache/hbase/pull/3210#issuecomment-829588567 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 34s | master passed | | +1 :green_heart: | compile | 1m 12s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 25s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | +1 :green_heart: | shadedjars | 8m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 141m 25s | hbase-server in the patch failed. | | | | 173m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3210 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7459d0f4dee3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/testReport/ | | Max. process+thread count | 4506 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3211: HBASE-25816: Improve the documentation of Architecture section of reference guide
Apache-HBase commented on pull request #3211: URL: https://github.com/apache/hbase/pull/3211#issuecomment-829583990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Kota-SH opened a new pull request #3211: HBASE-25816: Improve the documentation of Architecture section of reference guide
Kota-SH opened a new pull request #3211: URL: https://github.com/apache/hbase/pull/3211 This PR refers to [https://issues.apache.org/jira/browse/HBASE-25816](url) While going through the reference guide, I found these small changes that can be made to the documentation of the "Architecture" part of it. Hence I'm filing this Jira to track those changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] shahrs87 commented on pull request #3027: HBASE-25612 HMaster should abort if ReplicationLogCleaner encounters …
shahrs87 commented on pull request #3027: URL: https://github.com/apache/hbase/pull/3027#issuecomment-829572723 This bug doesn't exist in master branch. We share the same zk session as hmaster. So if there is any zk issue then master will call Abortable and shutdown master process. In branch-1 ReplicationLogCleaner creates its own zk session so it has to manage locally to abort master incase it encounters zk issue. Closing this PR and will create a patch for branch-1 soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] shahrs87 commented on pull request #3210: HBASE-25827 Per Cell TTL tags get duplicated with increments causing tags length overflow
shahrs87 commented on pull request #3210: URL: https://github.com/apache/hbase/pull/3210#issuecomment-829531565 @anoopsjohn @saintstack could you please review the patch ? Thank you ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335785#comment-17335785 ] Sean Busbey commented on HBASE-25524: - If we want the ability to confirm that a RS _can_ do SCR independent of the specific regions it is hosting then I think we need to use a canary file. I think #2 above is the only practical way to do that. Presumably we only need to do this once per RS process. I don't think it matters that later readings of hfiles might not have a local block to use for a SCR; that notion is captured in locality and scr metrics. After writing the canary file I don't see a straight forward way to check that HDFS wrote the first block locally, but we can use the FSDataInputStreamWrapper we maintain to read the file and then check if the read happened via SCR / zero copy > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3210: HBASE-25827 Per Cell TTL tags get duplicated with increments causing tags length overflow
Apache-HBase commented on pull request #3210: URL: https://github.com/apache/hbase/pull/3210#issuecomment-829503900 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 3m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 5s | master passed | | +1 :green_heart: | spotbugs | 2m 3s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 3m 17s | the patch passed | | +1 :green_heart: | javac | 3m 17s | the patch passed | | +1 :green_heart: | checkstyle | 1m 7s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 0s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 50m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3210 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux ed74df2a88c3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3210/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25524) Region Servers should give some indication that short-circuit reads are configured properly and working
[ https://issues.apache.org/jira/browse/HBASE-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335755#comment-17335755 ] Nick Dimiduk commented on HBASE-25524: -- Writing and reading back a temporary file seems like a good strategy. Do we have any introspection into the stream at all, to know if it's using the local file handle? > Region Servers should give some indication that short-circuit reads are > configured properly and working > --- > > Key: HBASE-25524 > URL: https://issues.apache.org/jira/browse/HBASE-25524 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver, Usability >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Rahul Kumar >Priority: Major > > Going through the exercise of confirming short-circuit reads turns out to be > quite a chore. 2.3.0 added a new metric, but metrics only get tickled when > there's traffic that happens to use the feature. This is not very convenient > for an operator who has taken a cluster out of service for maintenance. > It would be nice if the rs process could perform some test and emitted some > confirmation of the status of this otherwise transparent feature. After > confirmation, it could light up a boolean metric value and/or add mention of > it to the info page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ndimiduk commented on a change in pull request #3139:
URL: https://github.com/apache/hbase/pull/3139#discussion_r623288418
##
File path: hbase-common/src/main/resources/hbase-default.xml
##
@@ -638,12 +638,6 @@ possible configurations would overwhelm and obscure the
important.
true
Whether to merge a region as part of
normalization.
-
-hbase.normalizer.min.region.count
Review comment:
What's the reasoning for removing the default value from this file?
While it's redundant to use in code, this file serves as a source of
documentation ; I believe it is parsed by our site building process to populate
a section of the online book.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -101,11 +106,21 @@ public void onConfigurationChange(Configuration conf) {
setConf(conf);
}
- private static int parseMinRegionCount(final Configuration conf) {
-final int parsedValue = conf.getInt(MIN_REGION_COUNT_KEY,
DEFAULT_MIN_REGION_COUNT);
+ private static int parseMergeMinRegionCount(final Configuration conf) {
+String parsedStringValue = conf.get(MERGE_MIN_REGION_COUNT_KEY);
Review comment:
I _believe_ you can register the old key via the [deprecation
API](https://hadoop.apache.org/docs/r2.10.1/api/org/apache/hadoop/conf/Configuration.html#addDeprecation(java.lang.String,%20java.lang.String,%20java.lang.String))
and only ever access the new key from code. I believe this has the benefit of
only logging the warning once, instead of every time the implementation happens
to do a lookup.
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
##
@@ -370,6 +371,35 @@ public void testHonorsMergeEnabledInTD() {
@Test
public void testHonorsMinimumRegionCount() {
+conf.setInt(MERGE_MIN_REGION_COUNT_KEY, 1);
Review comment:
Can you instead keep a single implementation in a helper method that
allows for the configuration used to be passed in as a parameter? This would be
better than copy-pasting the unit test method body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25819) Fix style issues for StochasticLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-25819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335732#comment-17335732
]
Hudson commented on HBASE-25819:
Results for branch branch-2
[build #238 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix style issues for StochasticLoadBalancer
> ---
>
> Key: HBASE-25819
> URL: https://issues.apache.org/jira/browse/HBASE-25819
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> There are some warnings when moving it to hbase-balancer module.
> Let's fix them before doing the actual moving as it could also by applied to
> branch-2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25779) HRegionServer#compactSplitThread should be private
[
https://issues.apache.org/jira/browse/HBASE-25779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335731#comment-17335731
]
Hudson commented on HBASE-25779:
Results for branch branch-2
[build #238 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/238/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HRegionServer#compactSplitThread should be private
> --
>
> Key: HBASE-25779
> URL: https://issues.apache.org/jira/browse/HBASE-25779
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Trivial
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Minor refactor. Make the {{compactSplitThread}} member field of
> {{HRegionServer}} private, and gate all access through the getter method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] shahrs87 opened a new pull request #3210: HBASE-25827 Per Cell TTL tags get duplicated with increments causing tags length overflow
shahrs87 opened a new pull request #3210: URL: https://github.com/apache/hbase/pull/3210 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-25794) Fix checkstyle violations in hbase-table-reporter module
[
https://issues.apache.org/jira/browse/HBASE-25794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shanmukha Haripriya Kota reassigned HBASE-25794:
Assignee: Shanmukha Haripriya Kota
> Fix checkstyle violations in hbase-table-reporter module
>
>
> Key: HBASE-25794
> URL: https://issues.apache.org/jira/browse/HBASE-25794
> Project: HBase
> Issue Type: Task
> Components: hbase-operator-tools
>Reporter: Peter Somogyi
>Assignee: Shanmukha Haripriya Kota
>Priority: Trivial
> Labels: beginner
>
> The hbase-table-reporter module in hbase-operator-tools has a bunch of
> checkstyle violations.
> {noformat}
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[22]
> (imports) ImportOrder: Wrong order for 'org.apache.commons.cli.Option' import.
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[23]
> (imports) ImportOrder: Wrong order for
> 'org.apache.commons.cli.CommandLineParser' import.
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[24]
> (imports) ImportOrder: Wrong order for 'org.apache.commons.cli.CommandLine'
> import.
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[34,8]
> (imports) UnusedImports: Unused import - org.apache.hadoop.hbase.HRegionInfo.
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[37]
> (imports) AvoidStarImport: Using the '.*' form of import should be avoided -
> org.apache.hadoop.hbase.client.*.
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[41]
> (imports) ImportOrder: Wrong order for 'java.io.BufferedWriter' import.
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[66,1]
> (design) HideUtilityClassConstructor: Utility classes should not have a
> public or default constructor.
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[94]
> (sizes) LineLength: Line is longer than 100 characters (found 120).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[126]
> (sizes) LineLength: Line is longer than 100 characters (found 105).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[132]
> (sizes) LineLength: Line is longer than 100 characters (found 106).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[247]
> (indentation) Indentation: 'throws' has incorrect indentation level 2,
> expected level should be 4.
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[275]
> (sizes) LineLength: Line is longer than 100 characters (found 112).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[279]
> (sizes) LineLength: Line is longer than 100 characters (found 105).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[308]
> (sizes) LineLength: Line is longer than 100 characters (found 101).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[370]
> (sizes) LineLength: Line is longer than 100 characters (found 121).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[371]
> (sizes) LineLength: Line is longer than 100 characters (found 108).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[375]
> (sizes) LineLength: Line is longer than 100 characters (found 102).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[401]
> (sizes) LineLength: Line is longer than 100 characters (found 113).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[407]
> (sizes) LineLength: Line is longer than 100 characters (found 112).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[410]
> (sizes) LineLength: Line is longer than 100 characters (found 107).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[437]
> (sizes) LineLength: Line is longer than 100 characters (found 114).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[438]
> (sizes) LineLength: Line is longer than 100 characters (found 109).
> [ERROR] src/main/java/org/apache/hbase/reporter/TableReporter.java:[445]
> (indentation) Indentation: 'throws' has incorrect indentation level 2,
> expected level should be 4.
> [ERROR] src/test/java/org/apache/hbase/reporter/TestTableReporter.java:[27]
> (imports) ImportOrder: Wrong order for 'java.util.ArrayList' import.
> [ERROR] src/test/java/org/apache/hbase/reporter/TestTableReporter.java:[30]
> (imports) ImportOrder: Wrong order for
> 'org.apache.hadoop.hbase.shaded.junit.framework.TestCase.assertEquals' import.
> [ERROR] src/test/java/org/apache/hbase/reporter/TestTableReporter.java:[59]
> (sizes) LineLength: Line is longer than 100 characters (found 110).
> [ERROR]
[jira] [Commented] (HBASE-25827) Per Cell TTL tags get duplicated with increments causing tags length overflow
[
https://issues.apache.org/jira/browse/HBASE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335551#comment-17335551
]
Rushabh Shah commented on HBASE-25827:
--
I have test case to reproduce this issue. I am working on the patch.
> Per Cell TTL tags get duplicated with increments causing tags length overflow
> -
>
> Key: HBASE-25827
> URL: https://issues.apache.org/jira/browse/HBASE-25827
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.1.9, 2.2.6
>Reporter: Shashank Thillainathan
>Assignee: Rushabh Shah
>Priority: Critical
>
> Incrementing with per cell TTL and flushing corrupts the HFile.
>
> Reproducing the issue:
> Incrementing a row and a column with per cell TTL for about 3 thousand times
> and flushing corrupts the HFile leaving the table unusable.
> Cause:
> On reading the HFile, it is found that duplicate TTL tags get appended for
> each cell.
> Though this case has already been addressed here at HBASE-18030, corruption
> still occurs even with this patch.
> {code:java}
> java.lang.IllegalStateException: Invalid currTagsLen -32767. Block offset:
> 16665, block length: 65596, position: 0 (without header).
> path=hdfs://hdfs/file/path
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkTagsLen(HFileReaderImpl.java:642)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:630)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.updateCurrentBlock(HFileReaderImpl.java:1206)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.loadBlockAndSeekToKey(HFileReaderImpl.java:1149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:863)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:837)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:347)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:256)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:469)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:369)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:311)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:275)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:1038)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:1029)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:764)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:695)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6593)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6757)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6527)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6504)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6491)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7458)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7436)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:8123)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.reckonDeltasByStore(HRegion.java:8003)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.reckonDeltas(HRegion.java:7958)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doDelta(HRegion.java:7805)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:7767)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:734)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2705)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42290)
> at
[jira] [Assigned] (HBASE-25827) Per Cell TTL tags get duplicated with increments causing tags length overflow
[
https://issues.apache.org/jira/browse/HBASE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah reassigned HBASE-25827:
Assignee: Rushabh Shah
> Per Cell TTL tags get duplicated with increments causing tags length overflow
> -
>
> Key: HBASE-25827
> URL: https://issues.apache.org/jira/browse/HBASE-25827
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.1.9, 2.2.6
>Reporter: Shashank Thillainathan
>Assignee: Rushabh Shah
>Priority: Critical
>
> Incrementing with per cell TTL and flushing corrupts the HFile.
>
> Reproducing the issue:
> Incrementing a row and a column with per cell TTL for about 3 thousand times
> and flushing corrupts the HFile leaving the table unusable.
> Cause:
> On reading the HFile, it is found that duplicate TTL tags get appended for
> each cell.
> Though this case has already been addressed here at HBASE-18030, corruption
> still occurs even with this patch.
> {code:java}
> java.lang.IllegalStateException: Invalid currTagsLen -32767. Block offset:
> 16665, block length: 65596, position: 0 (without header).
> path=hdfs://hdfs/file/path
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkTagsLen(HFileReaderImpl.java:642)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:630)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.updateCurrentBlock(HFileReaderImpl.java:1206)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.loadBlockAndSeekToKey(HFileReaderImpl.java:1149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:863)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:837)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:347)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:256)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:469)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:369)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:311)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:275)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:1038)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:1029)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:764)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:695)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6593)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6757)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6527)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6504)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6491)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7458)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7436)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:8123)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.reckonDeltasByStore(HRegion.java:8003)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.reckonDeltas(HRegion.java:7958)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doDelta(HRegion.java:7805)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:7767)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:734)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2705)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42290)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at
[GitHub] [hbase] Reidddddd merged pull request #3193: [branch-1] Make hbase-thrift module build with jdk8
Reidd merged pull request #3193: URL: https://github.com/apache/hbase/pull/3193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25790) NamedQueue 'BalancerRejection' for recent history of balancer skipping
[
https://issues.apache.org/jira/browse/HBASE-25790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated HBASE-25790:
-
Fix Version/s: 2.4.3
2.5.0
3.0.0-alpha-1
> NamedQueue 'BalancerRejection' for recent history of balancer skipping
> --
>
> Key: HBASE-25790
> URL: https://issues.apache.org/jira/browse/HBASE-25790
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, master
>Reporter: Zhuoyue Huang
>Assignee: Zhuoyue Huang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> [HBASE-24528|https://issues.apache.org/jira/browse/HBASE-24528] implements a
> NamedQueue that saved recent history of balancer decision details. Since we
> want to know exactly why the balancer is skipping to run, we need a new
> namedQueue called 'BalancerRejection' to hold the history of all details
> information why the balancer was rejected to run(skipping). LogEntry of
> 'BalancerRejection' should contain the information below:
> 1. Reason, to show why this time balancer was rejected
> Maybe look like:
>
> {code:java}
> (cost1*multiplier1+cost2*multiplier2+...+costn*multipliern)/sumMultiplier=0.025
> <= minCostNeedBalance(0.35)
> {code}
> 2. Detail cost factors and multipliers
> Maybe look like:
> {code:java}
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$RegionCountSkewCostFunction
> cost:0.0 multiplier:500.0
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$MoveCostFunction
> cost:0.0 multiplier:7.0
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$RackLocalityCostFunction
> cost:1.0 multiplier:15.0
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
virajjasani commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-829298524 @GeorryHuang Before I commit, can you once execute new shell command locally and provide a screenshot of how result looks like? Please build latest code as per this PR. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
Apache-HBase commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-829289076 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | master passed | | +1 :green_heart: | compile | 2m 34s | master passed | | +1 :green_heart: | shadedjars | 8m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 12s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 2m 31s | the patch passed | | +1 :green_heart: | javac | 2m 31s | the patch passed | | +1 :green_heart: | shadedjars | 8m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 9s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 333m 25s | root in the patch failed. | | | | 374m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3182 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b007cab8cf6d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/testReport/ | | Max. process+thread count | 5064 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server hbase-shell . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829245219 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | compile | 2m 33s | master passed | | +1 :green_heart: | shadedjars | 8m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 24s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 2m 32s | the patch passed | | +1 :green_heart: | javac | 2m 32s | the patch passed | | +1 :green_heart: | shadedjars | 8m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 357m 16s | root in the patch failed. | | | | 395m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b0cc8bd1cfe2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/testReport/ | | Max. process+thread count | 5995 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
Apache-HBase commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-829243244 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 5s | master passed | | +1 :green_heart: | compile | 3m 46s | master passed | | +1 :green_heart: | shadedjars | 9m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 58s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 17s | the patch passed | | +1 :green_heart: | compile | 3m 39s | the patch passed | | +1 :green_heart: | javac | 3m 39s | the patch passed | | +1 :green_heart: | shadedjars | 9m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 262m 58s | root in the patch passed. | | | | 317m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3182 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6ad8098b23b3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/testReport/ | | Max. process+thread count | 4660 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server hbase-shell . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25819) Fix style issues for StochasticLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-25819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335479#comment-17335479
]
Hudson commented on HBASE-25819:
Results for branch master
[build #278 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/278/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/278/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/278/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/278/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix style issues for StochasticLoadBalancer
> ---
>
> Key: HBASE-25819
> URL: https://issues.apache.org/jira/browse/HBASE-25819
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> There are some warnings when moving it to hbase-balancer module.
> Let's fix them before doing the actual moving as it could also by applied to
> branch-2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25779) HRegionServer#compactSplitThread should be private
[
https://issues.apache.org/jira/browse/HBASE-25779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335478#comment-17335478
]
Hudson commented on HBASE-25779:
Results for branch master
[build #278 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/278/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/278/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/278/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/278/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HRegionServer#compactSplitThread should be private
> --
>
> Key: HBASE-25779
> URL: https://issues.apache.org/jira/browse/HBASE-25779
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Trivial
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Minor refactor. Make the {{compactSplitThread}} member field of
> {{HRegionServer}} private, and gate all access through the getter method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3209: HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should cha…
Apache-HBase commented on pull request #3209: URL: https://github.com/apache/hbase/pull/3209#issuecomment-829200328 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 0s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 9m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | the patch passed | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | +1 :green_heart: | shadedjars | 9m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 207m 13s | hbase-server in the patch passed. | | | | 242m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3209/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3209 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux eae7d8fc541a 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3209/1/testReport/ | | Max. process+thread count | 3460 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3209/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829199223 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 40s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 55s | master passed | | +1 :green_heart: | compile | 3m 44s | master passed | | +1 :green_heart: | shadedjars | 10m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 23s | the patch passed | | +1 :green_heart: | compile | 3m 47s | the patch passed | | +1 :green_heart: | javac | 3m 47s | the patch passed | | +1 :green_heart: | shadedjars | 9m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 277m 4s | root in the patch passed. | | | | 329m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0e2cde2416f9 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/testReport/ | | Max. process+thread count | 4382 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-829196409 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 44s | master passed | | +1 :green_heart: | compile | 1m 38s | master passed | | +1 :green_heart: | shadedjars | 8m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 44s | the patch passed | | +1 :green_heart: | compile | 1m 39s | the patch passed | | +1 :green_heart: | javac | 1m 39s | the patch passed | | +1 :green_heart: | shadedjars | 9m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | hbase-balancer generated 0 new + 2 unchanged - 2 fixed = 2 total (was 4) | | +1 :green_heart: | javadoc | 0m 40s | hbase-server in the patch passed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 49s | hbase-balancer in the patch passed. | | -1 :x: | unit | 201m 9s | hbase-server in the patch failed. | | | | 244m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ec83d7b1adc0 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/testReport/ | | Max. process+thread count | 2974 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3209: HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should cha…
Apache-HBase commented on pull request #3209: URL: https://github.com/apache/hbase/pull/3209#issuecomment-829154542 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 8m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 8m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 152m 15s | hbase-server in the patch passed. | | | | 183m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3209/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3209 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8736adcf8d10 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3209/1/testReport/ | | Max. process+thread count | 4896 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3209/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-829148866 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 8m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | hbase-balancer generated 0 new + 1 unchanged - 2 fixed = 1 total (was 3) | | +1 :green_heart: | javadoc | 0m 37s | hbase-server in the patch passed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 53s | hbase-balancer in the patch passed. | | -1 :x: | unit | 144m 33s | hbase-server in the patch failed. | | | | 180m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3d774e291730 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/testReport/ | | Max. process+thread count | 4129 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25827) Per Cell TTL tags get duplicated with increments causing tags length overflow
[
https://issues.apache.org/jira/browse/HBASE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashank Thillainathan updated HBASE-25827:
---
Description:
Incrementing with per cell TTL and flushing corrupts the HFile.
Reproducing the issue:
Incrementing a row and a column with per cell TTL for about 3 thousand times
and flushing corrupts the HFile leaving the table unusable.
Cause:
On reading the HFile, it is found that duplicate TTL tags get appended for
each cell.
Though this case has already been addressed here at HBASE-18030, corruption
still occurs even with this patch.
{code:java}
java.lang.IllegalStateException: Invalid currTagsLen -32767. Block offset:
16665, block length: 65596, position: 0 (without header).
path=hdfs://hdfs/file/path
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkTagsLen(HFileReaderImpl.java:642)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:630)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.updateCurrentBlock(HFileReaderImpl.java:1206)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.loadBlockAndSeekToKey(HFileReaderImpl.java:1149)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:863)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:837)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:347)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:256)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:469)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:369)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:311)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:275)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:1038)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:1029)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:764)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:695)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6593)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6757)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6527)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6504)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6491)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7458)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7436)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:8123)
at
org.apache.hadoop.hbase.regionserver.HRegion.reckonDeltasByStore(HRegion.java:8003)
at
org.apache.hadoop.hbase.regionserver.HRegion.reckonDeltas(HRegion.java:7958)
at
org.apache.hadoop.hbase.regionserver.HRegion.doDelta(HRegion.java:7805)
at
org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:7767)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:734)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2705)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42290)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
was:
Incrementing with per cell TTL and flushing corrupts the HFile.
Reproducing the issue:
Incrementing a row and a column with per cell TTL for about 3 thousand times
and flushing corrupts the HFile leaving the table unusable.
Cause:
On reading the HFile, it is found that duplicate TTL tags get appended for
each cell.
Though this case has already been addressed here at HBASE-18030, corruption
still occurs even with this patch.
{code:java}
[jira] [Updated] (HBASE-25827) Per Cell TTL tags get duplicated with increments causing tags length overflow
[
https://issues.apache.org/jira/browse/HBASE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashank Thillainathan updated HBASE-25827:
---
Description:
Incrementing with per cell TTL and flushing corrupts the HFile.
Reproducing the issue:
Incrementing a row and a column with per cell TTL for about 3 thousand times
and flushing corrupts the HFile leaving the table unusable.
Cause:
On reading the HFile, it is found that duplicate TTL tags get appended for
each cell.
Though this case has already been addressed here at HBASE-18030, corruption
still occurs even with this patch.
{code:java}
java.lang.IllegalStateException: Invalid currTagsLen -31260. Block offset:
250962, block length: 76568, position: 42207 (without header).
path=hdfs://hdfs/file/path
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkTagsLen(HFileReaderImpl.java:642)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:630)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1080)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:388)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1432)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2192)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:577)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:619)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
Incrementing with per cell TTL and flushing corrupts the HFile.
Reproducing the issue:
Incrementing a row and a column with per cell TTL for about 3 thousand times
and flushing corrupts the HFile leaving the table unusable.
Cause:
On reading the HFile, it is found that duplicate TTL tags get appended for
each cell.
Though this case has already been addressed here at HBASE-18030, corruption
still occurs even with this patch.
{code:java}
java.lang.IllegalStateException: Invalid currTagsLen -31260. Block offset:
250962, block length: 76568, position: 42207 (without header).
path=hdfs://hdfs/file/path
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkTagsLen(HFileReaderImpl.java:642)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:630)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1080)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:388)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1432)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2192)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:577)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:619)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
[jira] [Updated] (HBASE-25827) Per Cell TTL tags get duplicated with increments causing tags length overflow
[
https://issues.apache.org/jira/browse/HBASE-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashank Thillainathan updated HBASE-25827:
---
Description:
Incrementing with per cell TTL and flushing corrupts the HFile.
Reproducing the issue:
Incrementing a row and a column with per cell TTL for about 3 thousand times
and flushing corrupts the HFile leaving the table unusable.
Cause:
On reading the HFile, it is found that duplicate TTL tags get appended for
each cell.
Though this case has already been addressed here at HBASE-18030, corruption
still occurs even with this patch.
{code:java}
java.lang.IllegalStateException: Invalid currTagsLen -31260. Block offset:
250962, block length: 76568, position: 42207 (without header).
path=hdfs://hdfs/file/path
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkTagsLen(HFileReaderImpl.java:642)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:630)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1080)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:388)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1432)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2192)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:577)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:619)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
Incrementing with per cell TTL and flushing corrupts the HFile.
Reproducing the issue:
Incrementing a row and a column with per cell TTL for about 3 thousand times
and flushing corrupts the HFile leaving the table unusable.
Cause:
On reading the HFile, it is found that duplicate TTL tags get appended for each
cell.
Though this case has already been addressed here at
[HBASE-18030|https://issues.apache.org/jira/browse/HBASE-18030], corruption
still occurs even with this patch.
{code:java}
// java.lang.IllegalStateException: Invalid currTagsLen -31260. Block offset:
250962, block length: 76568, position: 42207 (without header).
path=hdfs://hdfs/file/path
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkTagsLen(HFileReaderImpl.java:642)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:630)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1080)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:388)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1432)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2192)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:577)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:619)
at
[jira] [Created] (HBASE-25827) Per Cell TTL tags get duplicated with increments causing tags length overflow
Shashank Thillainathan created HBASE-25827:
--
Summary: Per Cell TTL tags get duplicated with increments causing
tags length overflow
Key: HBASE-25827
URL: https://issues.apache.org/jira/browse/HBASE-25827
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.2.6, 2.1.9
Reporter: Shashank Thillainathan
Incrementing with per cell TTL and flushing corrupts the HFile.
Reproducing the issue:
Incrementing a row and a column with per cell TTL for about 3 thousand times
and flushing corrupts the HFile leaving the table unusable.
Cause:
On reading the HFile, it is found that duplicate TTL tags get appended for each
cell.
Though this case has already been addressed here at
[HBASE-18030|https://issues.apache.org/jira/browse/HBASE-18030], corruption
still occurs even with this patch.
{code:java}
// java.lang.IllegalStateException: Invalid currTagsLen -31260. Block offset:
250962, block length: 76568, position: 42207 (without header).
path=hdfs://hdfs/file/path
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.checkTagsLen(HFileReaderImpl.java:642)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readKeyValueLen(HFileReaderImpl.java:630)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1080)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:388)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1432)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2192)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:577)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:619)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
Apache-HBase commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-829121747 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 9m 24s | master passed | | +1 :green_heart: | checkstyle | 2m 18s | master passed | | +0 :ok: | refguide | 3m 43s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 20m 53s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | compile | 11m 40s | the patch passed | | +1 :green_heart: | cc | 11m 40s | the patch passed | | +1 :green_heart: | javac | 11m 40s | the patch passed | | -0 :warning: | checkstyle | 2m 33s | root: The patch generated 17 new + 231 unchanged - 0 fixed = 248 total (was 231) | | -0 :warning: | rubocop | 0m 17s | The patch generated 4 new + 387 unchanged - 0 fixed = 391 total (was 387) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 4m 50s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 24m 15s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 8m 43s | the patch passed | | +1 :green_heart: | spotbugs | 25m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 33s | The patch does not generate ASF License warnings. | | | | 137m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3182 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile refguide xml cc hbaseprotoc prototool rubocop | | uname | Linux f1d2d82370b1 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/artifact/yetus-general-check/output/branch-site/book.html | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 126 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server hbase-shell . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/16/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3209: HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should cha…
Apache-HBase commented on pull request #3209: URL: https://github.com/apache/hbase/pull/3209#issuecomment-829075361 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | compile | 3m 12s | master passed | | +1 :green_heart: | checkstyle | 1m 5s | master passed | | +1 :green_heart: | spotbugs | 2m 6s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 3m 6s | the patch passed | | +1 :green_heart: | javac | 3m 6s | the patch passed | | +1 :green_heart: | checkstyle | 1m 2s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 47m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3209/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3209 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6b976adeff0d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3209/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-829074788 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 3m 47s | master passed | | +1 :green_heart: | checkstyle | 1m 22s | master passed | | +1 :green_heart: | spotbugs | 2m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 3m 48s | the patch passed | | -0 :warning: | javac | 0m 29s | hbase-balancer generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) | | +1 :green_heart: | checkstyle | 1m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 51s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 54m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux f0cbc153b498 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 opened a new pull request #3209: HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should cha…
Apache9 opened a new pull request #3209: URL: https://github.com/apache/hbase/pull/3209 …in the request to internal balancer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-829039707 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 8m 42s | master passed | | +1 :green_heart: | checkstyle | 2m 14s | master passed | | +1 :green_heart: | spotbugs | 11m 20s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 8m 41s | the patch passed | | -0 :warning: | javac | 8m 41s | root generated 4 new + 1557 unchanged - 4 fixed = 1561 total (was 1561) | | -0 :warning: | checkstyle | 2m 17s | root: The patch generated 373 new + 0 unchanged - 0 fixed = 373 total (was 0) | | -0 :warning: | whitespace | 0m 0s | The patch 1 line(s) with tabs. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 19m 53s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 11m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | | | | 83m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 9d4e8364290d 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/artifact/yetus-general-check/output/whitespace-tabs.txt | | Max. process+thread count | 126 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
Apache-HBase commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-829034004 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | compile | 8m 16s | master passed | | +1 :green_heart: | checkstyle | 1m 57s | master passed | | +0 :ok: | refguide | 3m 23s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 17m 22s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 53s | root in the patch failed. | | -1 :x: | compile | 4m 43s | root in the patch failed. | | -0 :warning: | cc | 4m 43s | root in the patch failed. | | -0 :warning: | javac | 4m 43s | root in the patch failed. | | -0 :warning: | checkstyle | 1m 57s | root: The patch generated 17 new + 231 unchanged - 0 fixed = 248 total (was 231) | | -0 :warning: | rubocop | 0m 13s | The patch generated 4 new + 387 unchanged - 0 fixed = 391 total (was 387) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 3m 19s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | -1 :x: | hadoopcheck | 2m 3s | The patch causes 16 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 4m 10s | The patch causes 16 errors with Hadoop v3.2.1. | | -1 :x: | hadoopcheck | 6m 19s | The patch causes 16 errors with Hadoop v3.3.0. | | -1 :x: | hbaseprotoc | 0m 33s | hbase-server in the patch failed. | | -1 :x: | hbaseprotoc | 2m 3s | root in the patch failed. | | -1 :x: | spotbugs | 0m 31s | hbase-server in the patch failed. | | -1 :x: | spotbugs | 5m 30s | root in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 17s | The patch does not generate ASF License warnings. | | | | 76m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3182 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile refguide xml cc hbaseprotoc prototool rubocop | | uname | Linux 9b4c57ee0a7d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-general-check/output/branch-site/book.html | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-general-check/output/patch-compile-root.txt | | cc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-general-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-general-check/output/patch-compile-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | refguide |
[jira] [Created] (HBASE-25826) Revisit the synchronization of balancer implementation
Duo Zhang created HBASE-25826: - Summary: Revisit the synchronization of balancer implementation Key: HBASE-25826 URL: https://issues.apache.org/jira/browse/HBASE-25826 Project: HBase Issue Type: Improvement Reporter: Duo Zhang In StochasticLoadBalancer, some methods are synchronized, for example, setClusterMetrics, setConf, balanceTable, etc. What I can see is that, we have ConfigurationObserver support, so we need to make sure that the configuration updating does not effect the balance operation, so we need synchronization here. But do we support two balance operations at the same time? And what about other balancer implementations? I think we need to make the synchronization more clearer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
Apache-HBase commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-829018989 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 14s | master passed | | +1 :green_heart: | compile | 3m 30s | master passed | | +1 :green_heart: | shadedjars | 9m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 2m 56s | root in the patch failed. | | -1 :x: | compile | 2m 24s | root in the patch failed. | | -0 :warning: | javac | 2m 24s | root in the patch failed. | | -1 :x: | shadedjars | 7m 31s | patch has 16 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 19s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 7m 37s | root in the patch failed. | | | | 54m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3182 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ff54d047d8a5 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/testReport/ | | Max. process+thread count | 693 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server hbase-shell . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-829015566 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 57s | master passed | | +1 :green_heart: | compile | 1m 38s | master passed | | +1 :green_heart: | shadedjars | 9m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 53s | the patch passed | | +1 :green_heart: | compile | 1m 43s | the patch passed | | +1 :green_heart: | javac | 1m 43s | the patch passed | | +1 :green_heart: | shadedjars | 9m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | hbase-balancer generated 0 new + 2 unchanged - 2 fixed = 2 total (was 4) | | +1 :green_heart: | javadoc | 0m 41s | hbase-server in the patch passed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 34s | hbase-balancer in the patch passed. | | -1 :x: | unit | 16m 48s | hbase-server in the patch failed. | | | | 65m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d15e4d260698 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/testReport/ | | Max. process+thread count | 653 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
Apache-HBase commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-829013087 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 3m 9s | master passed | | +1 :green_heart: | shadedjars | 8m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 12s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 53s | root in the patch failed. | | -1 :x: | compile | 1m 36s | root in the patch failed. | | -0 :warning: | javac | 1m 36s | root in the patch failed. | | -1 :x: | shadedjars | 6m 12s | patch has 16 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 6s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 7m 12s | root in the patch failed. | | | | 44m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3182 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bf5e5eaea9fa 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-root.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/testReport/ | | Max. process+thread count | 882 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-balancer hbase-server hbase-shell . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3182/15/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-829011581 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | compile | 1m 24s | master passed | | +1 :green_heart: | shadedjars | 9m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | shadedjars | 9m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | hbase-balancer generated 0 new + 1 unchanged - 2 fixed = 1 total (was 3) | | +1 :green_heart: | javadoc | 0m 38s | hbase-server in the patch passed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 11s | hbase-balancer in the patch passed. | | -1 :x: | unit | 17m 8s | hbase-server in the patch failed. | | | | 58m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c3a2c857c8fe 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/testReport/ | | Max. process+thread count | 531 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sunhelly commented on pull request #3025: HBASE-25517 Support parallel balance RSGroups
sunhelly commented on pull request #3025: URL: https://github.com/apache/hbase/pull/3025#issuecomment-829011480 Ping @Apache9 , is there any other concerns ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3206: HBASE-25818 Move StochasticLoadBalancer to hbase-balancer module
Apache-HBase commented on pull request #3206: URL: https://github.com/apache/hbase/pull/3206#issuecomment-829009577 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 3m 48s | master passed | | +1 :green_heart: | checkstyle | 1m 24s | master passed | | +1 :green_heart: | spotbugs | 2m 47s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | +1 :green_heart: | compile | 3m 46s | the patch passed | | -0 :warning: | javac | 0m 28s | hbase-balancer generated 6 new + 30 unchanged - 0 fixed = 36 total (was 30) | | -0 :warning: | checkstyle | 1m 11s | hbase-server: The patch generated 2 new + 91 unchanged - 0 fixed = 93 total (was 91) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 3s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 55m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3206 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5aceb539b5f3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c65314cdf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3206/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] GeorryHuang commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
GeorryHuang commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-829004018 Thanks @virajjasani for reviewing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sunhelly edited a comment on pull request #3025: HBASE-25517 Support parallel balance RSGroups
sunhelly edited a comment on pull request #3025: URL: https://github.com/apache/hbase/pull/3025#issuecomment-795406061 Thanks for reviewing, @Apache9. >Mind explaining a bit about the logic? And do you have any performance numbers? Yes, it's my pleasure. Currently, the balance for the whole cluster is, get all tables regions distribution map-> generate balance plans(by table or not) -> execute all plans by AM. As a result, for one RSGroup, how long the auto balance of it takes depending on how long the whole cluster plans generated and executed. For this patch, groups are balanced dependently, not only in generating plans, but also in executing plans. We propose this idea because our cluster was used a large multiplier of locality cost and before HBASE-18164 and HBASE-15515, some group balances were always stuck and blocked the other group balances, as well as the manually balances by shell commands, because they all used a whole cluster balancer lock. On our cluster with 20 groups, 1300+ tables, and 30+ regions, the balancer consumes about 3 minutes for a relevant balanced cluster with a very small number of regions to move or swap. While using parallelize balance with 10 fixed thread pool, it consumes about 1 minutes. For executing region plans, the duration is restricted in the max balancing time, which is configed by `hbase.balancer.max.balancing`. I think the executing time for the plans is not the most important here, but separating group balance locks and make them work without mutual influence is meaningful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] GeorryHuang removed a comment on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
GeorryHuang removed a comment on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-828994967 I guess this commit would cause unit tests to fail again. `[ERROR] Errors: [ERROR] TestFullBackupSet>TestBackupBase.tearDown:347 » NullPointer [ERROR] org.apache.hadoop.hbase.backup.TestFullBackupSet.testFullBackupSetExist [ERROR] Run 1: TestFullBackupSet>TestBackupBase.setUp:314 » YarnRuntime java.util.concurrent [ERROR] Run 2: TestFullBackupSet.testFullBackupSetExist:82 [ERROR] Run 3: TestFullBackupSet.testFullBackupSetExist:56 » IO org.apache.zookeeper.KeeperEx... [INFO] [ERROR] TestIncrementalBackupDeleteTable>TestBackupBase.tearDown:347 » NullPointer [ERROR] org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable.testIncBackupDeleteTable [ERROR] Run 1: TestIncrementalBackupDeleteTable>TestBackupBase.setUp:314 » YarnRuntime java.u... [ERROR] Run 2: TestIncrementalBackupDeleteTable.testIncBackupDeleteTable:95 » IO java.io.IOEx... [ERROR] Run 3: TestIncrementalBackupDeleteTable.testIncBackupDeleteTable:67 » IO org.apache.z... [INFO] [ERROR] org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore [ERROR] Run 1: TestIncrementalBackupMergeWithFailures>TestBackupBase.setUp:314 » YarnRuntime ... [ERROR] Run 2: TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore:263 » IO java... [ERROR] Run 3: TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore:236 » IO org [INFO] [ERROR] TestIncrementalBackupMergeWithFailures>TestBackupBase.tearDown:347 » NullPointer [ERROR] TestRemoteRestore>TestBackupBase.tearDown:347 » NullPointer [ERROR] org.apache.hadoop.hbase.backup.TestRemoteRestore.testFullRestoreRemote [ERROR] Run 1: TestRemoteRestore.setUp:45->TestBackupBase.setUp:314 » YarnRuntime java.util.c... [ERROR] Run 2: TestRemoteRestore.testFullRestoreRemote:61 » IllegalState Cannot restore hbase... [ERROR] Run 3: TestRemoteRestore.testFullRestoreRemote:57->TestBackupBase.backupTables:382 » IO [INFO] [ERROR] TestBackupLogCleaner>TestBackupBase.tearDown:347 » NullPointer [ERROR] org.apache.hadoop.hbase.backup.master.TestBackupLogCleaner.testBackupLogCleaner [ERROR] Run 1: TestBackupLogCleaner>TestBackupBase.setUp:314 » YarnRuntime java.util.concurre... [ERROR] Run 2: TestBackupLogCleaner.testBackupLogCleaner:133->TestBackupBase.backupTables:385 » IO [ERROR] Run 3: TestBackupLogCleaner.testBackupLogCleaner:75 » IO org.apache.zookeeper.KeeperE... [INFO] ` Can only reproduce those failure with CI, my local machine will passed all unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] GeorryHuang closed pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
GeorryHuang closed pull request #3182: URL: https://github.com/apache/hbase/pull/3182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] GeorryHuang commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
GeorryHuang commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-828994967 I guess this commit would cause unit tests to fail again. `[ERROR] Errors: [ERROR] TestFullBackupSet>TestBackupBase.tearDown:347 » NullPointer [ERROR] org.apache.hadoop.hbase.backup.TestFullBackupSet.testFullBackupSetExist [ERROR] Run 1: TestFullBackupSet>TestBackupBase.setUp:314 » YarnRuntime java.util.concurrent [ERROR] Run 2: TestFullBackupSet.testFullBackupSetExist:82 [ERROR] Run 3: TestFullBackupSet.testFullBackupSetExist:56 » IO org.apache.zookeeper.KeeperEx... [INFO] [ERROR] TestIncrementalBackupDeleteTable>TestBackupBase.tearDown:347 » NullPointer [ERROR] org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable.testIncBackupDeleteTable [ERROR] Run 1: TestIncrementalBackupDeleteTable>TestBackupBase.setUp:314 » YarnRuntime java.u... [ERROR] Run 2: TestIncrementalBackupDeleteTable.testIncBackupDeleteTable:95 » IO java.io.IOEx... [ERROR] Run 3: TestIncrementalBackupDeleteTable.testIncBackupDeleteTable:67 » IO org.apache.z... [INFO] [ERROR] org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore [ERROR] Run 1: TestIncrementalBackupMergeWithFailures>TestBackupBase.setUp:314 » YarnRuntime ... [ERROR] Run 2: TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore:263 » IO java... [ERROR] Run 3: TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore:236 » IO org [INFO] [ERROR] TestIncrementalBackupMergeWithFailures>TestBackupBase.tearDown:347 » NullPointer [ERROR] TestRemoteRestore>TestBackupBase.tearDown:347 » NullPointer [ERROR] org.apache.hadoop.hbase.backup.TestRemoteRestore.testFullRestoreRemote [ERROR] Run 1: TestRemoteRestore.setUp:45->TestBackupBase.setUp:314 » YarnRuntime java.util.c... [ERROR] Run 2: TestRemoteRestore.testFullRestoreRemote:61 » IllegalState Cannot restore hbase... [ERROR] Run 3: TestRemoteRestore.testFullRestoreRemote:57->TestBackupBase.backupTables:382 » IO [INFO] [ERROR] TestBackupLogCleaner>TestBackupBase.tearDown:347 » NullPointer [ERROR] org.apache.hadoop.hbase.backup.master.TestBackupLogCleaner.testBackupLogCleaner [ERROR] Run 1: TestBackupLogCleaner>TestBackupBase.setUp:314 » YarnRuntime java.util.concurre... [ERROR] Run 2: TestBackupLogCleaner.testBackupLogCleaner:133->TestBackupBase.backupTables:385 » IO [ERROR] Run 3: TestBackupLogCleaner.testBackupLogCleaner:75 » IO org.apache.zookeeper.KeeperE... [INFO] ` Can only reproduce those failure with CI, my local machine will passed all unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3208: HBASE-25824 IntegrationTestLoadCommonCrawl
Apache-HBase commented on pull request #3208: URL: https://github.com/apache/hbase/pull/3208#issuecomment-828988338 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | compile | 2m 43s | master passed | | +1 :green_heart: | shadedjars | 8m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 2m 51s | the patch passed | | +1 :green_heart: | javac | 2m 51s | the patch passed | | +1 :green_heart: | shadedjars | 8m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 405m 10s | root in the patch passed. | | | | 446m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3208 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 462d4704056e 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2382f68b23 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/1/testReport/ | | Max. process+thread count | 3934 (vs. ulimit of 3) | | modules | C: hbase-it . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3208/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25825) RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer
Duo Zhang created HBASE-25825: - Summary: RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer Key: HBASE-25825 URL: https://issues.apache.org/jira/browse/HBASE-25825 Project: HBase Issue Type: Bug Components: Balancer Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
Apache-HBase commented on pull request #3182: URL: https://github.com/apache/hbase/pull/3182#issuecomment-828982857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] GeorryHuang commented on a change in pull request #3182: HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping
GeorryHuang commented on a change in pull request #3182:
URL: https://github.com/apache/hbase/pull/3182#discussion_r622770226
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -4220,6 +4229,12 @@ private void getProcedureResult(long procId,
CompletableFuture future, int
"Balancer Decision logs are not maintained by HRegionServer");
}
return getBalancerDecisions(limit);
Review comment:
Okey
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/BalancerRejection.java
##
@@ -0,0 +1,117 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.client;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.util.GsonUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.yetus.audience.InterfaceStability;
+
+import org.apache.hbase.thirdparty.com.google.gson.Gson;
+import org.apache.hbase.thirdparty.com.google.gson.JsonSerializer;
+
+/**
+ * History of detail information that balancer movements was rejected
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+final public class BalancerRejection extends LogEntry {
+ //The reason why balancer was rejected
+ private final String reason;
+ private final List costFuncInfoList;
+
+ // used to convert object to pretty printed format
+ // used by toJsonPrettyPrint()
+ private static final Gson GSON = GsonUtil.createGson()
+.setPrettyPrinting()
+.disableHtmlEscaping()
+.registerTypeAdapter(BalancerRejection.class,
(JsonSerializer)
+ (balancerRejection, type, jsonSerializationContext) -> {
+Gson gson = new Gson();
+return gson.toJsonTree(balancerRejection);
+ }).create();
+
+ private BalancerRejection(String reason, List costFuncInfoList) {
+this.reason = reason;
+if(costFuncInfoList == null){
+ this.costFuncInfoList = Collections.emptyList();
+}
+else {
+ this.costFuncInfoList = costFuncInfoList;
+}
+ }
+
+ public String getReason() {
+return reason;
+ }
+
+ public List getCostFuncInfoList() {
+return costFuncInfoList;
+ }
+
+ @Override
+ public String toString() {
+return new ToStringBuilder(this)
+ .append("reason", reason)
+ .append("costFuncInfoList", costFuncInfoList.toString())
+ .toString();
+ }
+
+ @Override
+ public String toJsonPrettyPrint() {
+return GSON.toJson(this);
+ }
+
+ public static class Builder {
+private String reason;
+private List costFuncInfoList;
+
+public Builder setReason(String reason) {
+ this.reason = reason;
+ return this;
+}
+
+public Builder addCostFuncInfo(String funcName, double cost, float
multiplier){
Review comment:
Okey
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -351,6 +364,21 @@ protected boolean needsBalance(TableName tableName,
BalancerClusterState cluster
boolean balanced = total <= 0 || sumMultiplier <= 0 ||
(sumMultiplier > 0 && (total / sumMultiplier) < minCostNeedBalance);
+if(balanced && isBalancerRejectionRecording){
+ String reason;
+ if (total <= 0) {
+reason =
"(cost1*multiplier1)+(cost2*multiplier2)+...+(costn*multipliern) = " + total +
" <= 0";
+ } else if (sumMultiplier <= 0) {
+reason = "sumMultiplier = " + sumMultiplier + " <= 0";
+ } else if ((total / sumMultiplier) < minCostNeedBalance) {
+reason =
+
"[(cost1*multiplier1)+(cost2*multiplier2)+...+(costn*multipliern)]/sumMultiplier
= " + (total
+/ sumMultiplier) + " <= minCostNeedBalance(" + minCostNeedBalance
+ ")";
+ } else {
+reason = "Unknown reason, please check the code for bugs";
Review comment:
Right! This else is unnecessary
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/impl/BalancerRejectionQueueService.java
##
@@ -0,0 +1,138 @@
+/*
+ *
+ * Licensed to the Apache
[GitHub] [hbase] ZhaoBQ commented on pull request #3139: HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.coun…
ZhaoBQ commented on pull request #3139: URL: https://github.com/apache/hbase/pull/3139#issuecomment-828969307 Ping @ndimiduk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
