[jira] [Updated] (HBASE-20396) Remove redundant MBean from thrift JMX
[ https://issues.apache.org/jira/browse/HBASE-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-20396: -- Resolution: Fixed Fix Version/s: 2.2.0 3.0.0 Status: Resolved (was: Patch Available) > Remove redundant MBean from thrift JMX > -- > > Key: HBASE-20396 > URL: https://issues.apache.org/jira/browse/HBASE-20396 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20396.master.001.patch > > > HBase has two types of thrift server. When any type of thrift server is > started, the MBean of the two thrift servers will be registered which is > redundant. > For example, when starting thrift2, we only need to register MBean > "Hadoop:service=HBase,name=Thrift,sub=ThriftTwo" of thrift2 in JMX, there's > no need to register MBean "Hadoop:service=HBase,name=Thrift,sub=ThriftOne" of > thrift. Vice versa -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20396) Remove redundant MBean from thrift JMX
[ https://issues.apache.org/jira/browse/HBASE-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536520#comment-16536520 ] Guangxu Cheng commented on HBASE-20396: --- Pushed to master and branch-2. Thanks [~yuzhih...@gmail.com] for review. > Remove redundant MBean from thrift JMX > -- > > Key: HBASE-20396 > URL: https://issues.apache.org/jira/browse/HBASE-20396 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20396.master.001.patch > > > HBase has two types of thrift server. When any type of thrift server is > started, the MBean of the two thrift servers will be registered which is > redundant. > For example, when starting thrift2, we only need to register MBean > "Hadoop:service=HBase,name=Thrift,sub=ThriftTwo" of thrift2 in JMX, there's > no need to register MBean "Hadoop:service=HBase,name=Thrift,sub=ThriftOne" of > thrift. Vice versa -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20838) Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils
[
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536513#comment-16536513
]
Yu Li commented on HBASE-20838:
---
bq. Let me know if you want help on figuring out the precommit changes to do
this
Yes I'd like to help but need some instructions (smile). And will update the
javadoc of {{CommonFSUtils}} in the same commit (if possible).
> Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
> Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
>
> As per
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
> in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the
> test should be in TestCommonFSUtils
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20822) TestAsyncNonMetaRegionLocator is flakey
[
https://issues.apache.org/jira/browse/HBASE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536511#comment-16536511
]
Hadoop QA commented on HBASE-20822:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
26s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
54s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}122m
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20822 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930771/HBASE-20822.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 783cef6c31e8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3bca01854a |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
[jira] [Updated] (HBASE-20822) TestAsyncNonMetaRegionLocator is flakey
[ https://issues.apache.org/jira/browse/HBASE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20822: -- Attachment: HBASE-20822.patch > TestAsyncNonMetaRegionLocator is flakey > --- > > Key: HBASE-20822 > URL: https://issues.apache.org/jira/browse/HBASE-20822 > Project: HBase > Issue Type: Bug > Components: asyncclient >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20822.patch, HBASE-20822.patch, > org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator-output.txt > > > This is a critical one, the error is result mismatch. Need to find out why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[ https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536427#comment-16536427 ] Ted Yu commented on HBASE-20836: lgtm > Add Yetus annotation for ReadReplicaClustersTableNameUtil > - > > Key: HBASE-20836 > URL: https://issues.apache.org/jira/browse/HBASE-20836 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: HBASE-18477 > > Attachments: HBASE-20836.HBASE-18477.001.patch, > HBASE-20836.HBASE-18477.002.patch > > > Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305
[
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536385#comment-16536385
]
Ted Yu commented on HBASE-20257:
[~busbey]:
Can you take another look ?
> hbase-spark should not depend on com.google.code.findbugs.jsr305
>
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
> Issue Type: Task
> Components: build, spark
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-20257.v01.patch, HBASE-20257.v02.patch,
> HBASE-20257.v03.patch, HBASE-20257.v04.patch
>
>
> The following can be observed in the build output of master branch:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed
> with message:
> We don't allow the JSR305 jar from the Findbugs project, see HBASE-16321.
> Found Banned Dependency: com.google.code.findbugs:jsr305:jar:1.3.9
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> Here is related snippet from hbase-spark/pom.xml:
> {code}
>
> com.google.code.findbugs
> jsr305
> {code}
> Dependency on jsr305 should be dropped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20578) Support region server group in target cluster
[ https://issues.apache.org/jira/browse/HBASE-20578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536384#comment-16536384 ] Ted Yu commented on HBASE-20578: Albert: Is there any issue that blocks you ? Thanks > Support region server group in target cluster > - > > Key: HBASE-20578 > URL: https://issues.apache.org/jira/browse/HBASE-20578 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Albert Lee >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20578-001.patch > > > When source tables belong to non-default region server group(s) and there are > region server group counterpart in the target cluster, we should support > replicating to target cluster using the region server group mapping. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536281#comment-16536281
]
Hudson commented on HBASE-18477:
Results for branch HBASE-18477
[build #258 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/258/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/258//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/258//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/258//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/258//console].
> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
> Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a
> root directory external to the cluster (such as in Amazon S3). This means
> that the data is stored outside of the cluster and can be accessible after
> the cluster has been terminated. One use case that is often asked about is
> pointing multiple clusters to one root directory (sharing the data) to have
> read resiliency in the case of a cluster failure.
>
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a
> read-replica HBase cluster that is pointed at the same root directory.
>
> This requires making the Read-Replica cluster Read-Only (no metadata
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets
> confused with multiple clusters trying to update the meta table with their ip
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data
> is picked up on the read replica cluster.
>
> This can be used with any existing cluster that is backed by an external
> filesystem.
>
> Please note that this feature is still quite manual (with the potential for
> automation later).
>
> More information on this particular feature can be found here:
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20842) Infinite loop when replaying remote wals
[
https://issues.apache.org/jira/browse/HBASE-20842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536113#comment-16536113
]
Hudson commented on HBASE-20842:
Results for branch master
[build #390 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/390/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/390//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/390//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/390//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Infinite loop when replaying remote wals
>
>
> Key: HBASE-20842
> URL: https://issues.apache.org/jira/browse/HBASE-20842
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20842.master.001.patch,
> HBASE-20842.master.002.patch, HBASE-20842.master.002.patch,
> HBASE-20842.master.002.patch
>
>
> {noformat}
> 2018-07-03 12:25:11,375 WARN [RSProcedureDispatcher-pool13-t19]
> replication.SyncReplicationReplayWALRemoteProcedure(107): Replay wals
> [remoteWALs/1-replay/asf916.gq1.ygridcore.net%2C36931%2C1530620616106-1530620683061-1.1530620683075.syncrep]
> on asf916.gq1.ygridcore.net,33811,1530620636539 failed for peer id=1
> org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server
> asf916.gq1.ygridcore.net,33811,1530620636539 is not online
> at
> org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$DeadRSRemoteCall.call(RSProcedureDispatcher.java:285)
> at
> org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$DeadRSRemoteCall.call(RSProcedureDispatcher.java:276)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-07-03 12:25:11,440 DEBUG [Thread-2883]
> replication.TestSyncReplicationStandbyKillRS(111): Server
> [asf916.gq1.ygridcore.net,33811,1530620636539] marked as dead, waiting for it
> to finish dead processing
> 2018-07-03 12:25:11,441 DEBUG [Thread-2883]
> replication.TestSyncReplicationStandbyKillRS(114): Server
> [asf916.gq1.ygridcore.net,33811,1530620636539] still being processed, waiting
> 2018-07-03 12:25:11,456 WARN [RS:3;asf916:45751] wal.AbstractFSWAL(419):
> 'hbase.regionserver.maxlogs' was deprecated.
> 2018-07-03 12:25:11,457 INFO [RS:3;asf916:45751] wal.AbstractFSWAL(424): WAL
> configuration: blocksize=256 MB, rollsize=128 MB,
> prefix=asf916.gq1.ygridcore.net%2C45751%2C1530620709275, suffix=,
> logDir=hdfs://localhost:42624/user/jenkins/test-data/a86a805e-162f-5f22-7b9e-573dbf0f40fb/WALs/asf916.gq1.ygridcore.net,45751,1530620709275,
>
> archiveDir=hdfs://localhost:42624/user/jenkins/test-data/a86a805e-162f-5f22-7b9e-573dbf0f40fb/oldWALs
> 2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-4]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
> handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1,
> datanodeId =
> DatanodeInfoWithStorage[127.0.0.1:38997,DS-6002160d-388b-4840-8538-e4c2255108be,DISK]
> 2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-5]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
> handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1,
> datanodeId =
> DatanodeInfoWithStorage[127.0.0.1:45904,DS-e189e3c8-a1bd-475c-86c0-3891e541fc6e,DISK]
> 2018-07-03 12:25:11,467 DEBUG [RS-EventLoopGroup-14-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping
> handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1,
> datanodeId =
> DatanodeInfoWithStorage[127.0.0.1:39589,DS-62ced3f8-35c4-4904-80cc-4d514b8f4050,DISK]
> 2018-07-03 12:25:11,495 DEBUG [RegionServerTracker-0]
> procedure2.ProcedureExecutor(887): Stored pid=30,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=asf916.gq1.ygridcore.net,33811,1530620636539, splitWal=true, meta=true
> 2018-07-03 12:25:11,495 DEBUG [RegionServerTracker-0]
>
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Status: Patch Available (was: Reopened) > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, > HBASE-20542.master.005-addendum.patch, run.sh, workloada, workloadc, > workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-20542: -- Attachment: HBASE-20542.master.005-addendum.patch > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, > HBASE-20542.master.005-addendum.patch, run.sh, workloada, workloadc, > workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20822) TestAsyncNonMetaRegionLocator is flakey
[ https://issues.apache.org/jira/browse/HBASE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536102#comment-16536102 ] Duo Zhang commented on HBASE-20822: --- [~zghaobac] [~openinx] PTAL. Thanks. > TestAsyncNonMetaRegionLocator is flakey > --- > > Key: HBASE-20822 > URL: https://issues.apache.org/jira/browse/HBASE-20822 > Project: HBase > Issue Type: Bug > Components: asyncclient >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20822.patch, > org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator-output.txt > > > This is a critical one, the error is result mismatch. Need to find out why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20822) TestAsyncNonMetaRegionLocator is flakey
[ https://issues.apache.org/jira/browse/HBASE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20822: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > TestAsyncNonMetaRegionLocator is flakey > --- > > Key: HBASE-20822 > URL: https://issues.apache.org/jira/browse/HBASE-20822 > Project: HBase > Issue Type: Bug > Components: asyncclient >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20822.patch, > org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator-output.txt > > > This is a critical one, the error is result mismatch. Need to find out why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20822) TestAsyncNonMetaRegionLocator is flakey
[ https://issues.apache.org/jira/browse/HBASE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20822: -- Attachment: HBASE-20822.patch > TestAsyncNonMetaRegionLocator is flakey > --- > > Key: HBASE-20822 > URL: https://issues.apache.org/jira/browse/HBASE-20822 > Project: HBase > Issue Type: Bug > Components: asyncclient >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20822.patch, > org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator-output.txt > > > This is a critical one, the error is result mismatch. Need to find out why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20822) TestAsyncNonMetaRegionLocator is flakey
[ https://issues.apache.org/jira/browse/HBASE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20822: -- Fix Version/s: 2.1.1 2.2.0 2.0.2 3.0.0 > TestAsyncNonMetaRegionLocator is flakey > --- > > Key: HBASE-20822 > URL: https://issues.apache.org/jira/browse/HBASE-20822 > Project: HBase > Issue Type: Bug > Components: asyncclient >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.2.0, 2.1.1 > > Attachments: HBASE-20822.patch, > org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator-output.txt > > > This is a critical one, the error is result mismatch. Need to find out why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20822) TestAsyncNonMetaRegionLocator is flakey
[
https://issues.apache.org/jira/browse/HBASE-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536097#comment-16536097
]
Duo Zhang commented on HBASE-20822:
---
OK the problem is in locateRowBeforeInCache...
As the netty thread may modify the meta cache after finishing the
CompletableFuture, so there maybe races that after we call clearCache in the
test, it then adds some entries into cache, and then cause the test to fail.
The problem for locateRowBeforeInCache is that, when the row we want to locate
is an empty array, then we just get the last entry, and the check is
{code}
if (isEmptyStopRow(loc.getRegion().getEndKey()) ||
Bytes.compareTo(loc.getRegion().getEndKey(), row) >= 0) {
{code}
Obviously if the row is a empty row then the second condition will always
pass...
Let me add a UT to reproduce it more easily and fix the problem.
> TestAsyncNonMetaRegionLocator is flakey
> ---
>
> Key: HBASE-20822
> URL: https://issues.apache.org/jira/browse/HBASE-20822
> Project: HBase
> Issue Type: Bug
> Components: asyncclient
>Reporter: Duo Zhang
>Priority: Major
> Attachments:
> org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator-output.txt
>
>
> This is a critical one, the error is result mismatch. Need to find out why.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
