[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369765#comment-16369765
]
Mike Drob commented on HBASE-19954:
---
Right, so is it the same thing happening? We start and stop a DFS cluster with
no region server start in between? I assume using a local file system is not an
option for the intent if the test.
Splitting into multiple classes is fine. Don't need 3 though, just 2 is enough.
Please add a comment to the one that doesn't use minihbase with a warning to
developers to not add tests that try to stay region server there later.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19978) The keepalive logic is incomplete in ProcedureExecutor
[ https://issues.apache.org/jira/browse/HBASE-19978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369725#comment-16369725 ] Hudson commented on HBASE-19978: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4615 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4615/]) HBASE-19978 The keepalive logic is incomplete in ProcedureExecutor (stack: rev 391790ddb05f0db0ffd67e0c2f2f961157af7be3) * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (add) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/DelayedProcedure.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestProcedurePriority.java * (add) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/InlineChore.java * (add) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/StoppableThread.java * (add) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/TimeoutExecutorThread.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java > The keepalive logic is incomplete in ProcedureExecutor > -- > > Key: HBASE-19978 > URL: https://issues.apache.org/jira/browse/HBASE-19978 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19978-v1.patch, HBASE-19978-v2.patch, > HBASE-19978.patch > > > The worker thread will just exit after keep alive time, and we never add it > back. The only way to add it back is through the stuck checker, this is not > correct. Here we should start new worker thread if it is under the core pool > size and there are pending procedures. > For now the default keep alive time is Long.MAX_VALUE which means no timeout > so no problem, but we do allow users to set it so we need to fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19293) Support adding a new replication peer in disabled state
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19293: - Summary: Support adding a new replication peer in disabled state (was: Add add a disabled state replication peer directly) > Support adding a new replication peer in disabled state > --- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch, HBASE-19293.master.003.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19293) Add add a disabled state replication peer directly
[ https://issues.apache.org/jira/browse/HBASE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19293: - Summary: Add add a disabled state replication peer directly (was: Support add a disabled state replication peer directly) > Add add a disabled state replication peer directly > --- > > Key: HBASE-19293 > URL: https://issues.apache.org/jira/browse/HBASE-19293 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19293.master.001.patch, > HBASE-19293.master.002.patch, HBASE-19293.master.003.patch > > > Now when add a replication peer, the default state is enabled. If you want > add a disabled replication peer, you need add a peer first, then disable it. > It need two step to finish now. > Use case for add a disabled replication peer. When user want sync data from a > cluster A to a new peer cluster. > 1. Add a disabled replication peer. And config the table to peer config. > 2. Take a snapshot of table and export snapshot to peer cluster. > 3. Restore snapshot in peer cluster. > 4. Enable the peer and wait all stuck replication log replicated to peer > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369688#comment-16369688
]
Hadoop QA commented on HBASE-19954:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
4s{color} | {color:red} hbase-server: The patch generated 1 new + 4 unchanged -
1 fixed = 5 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 52s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}103m
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19954 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911176/19954.v3.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d716038ecad3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 391790ddb0 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11574/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11574/testReport/ |
| Max.
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369672#comment-16369672
]
Hadoop QA commented on HBASE-19954:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
41s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
3s{color} | {color:red} hbase-server: The patch generated 1 new + 7 unchanged -
1 fixed = 8 total (was 8) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}135m
46s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19954 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911174/19954.v2.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 16e31f7bd63a 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a9a6eed372 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
[jira] [Updated] (HBASE-18794) Remove deprecated methods in MasterObserver
[ https://issues.apache.org/jira/browse/HBASE-18794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-18794: - Labels: incompatible (was: ) > Remove deprecated methods in MasterObserver > --- > > Key: HBASE-18794 > URL: https://issues.apache.org/jira/browse/HBASE-18794 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Labels: incompatible > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18794.v0.patch, HBASE-18794.v0.patch > > > Similar to HBASE-18973 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369649#comment-16369649
]
Ted Yu commented on HBASE-19954:
Mike:
I re-read HBASE-17922.
See my comment above on which patch added the shutdown hook clearing.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369643#comment-16369643
]
Mike Drob commented on HBASE-19954:
---
[~tedyu] - see the discussion on HBASE-17922 please. I suspect it is relevant.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369635#comment-16369635
]
Ted Yu commented on HBASE-19954:
In the @After method of TestBlockReorder, shutdownMiniCluster() is called.
Clearing shutdown hooks came from this patch:
https://issues.apache.org/jira/secure/attachment/12751805/HDFS-8838-HDFS-7285-20150821.patch#file-6
whose parent JIRA, HDFS-7285, was incorporated into hadoop3.
In patch v3, I separate the 3 subtests into 3 tests since they use different
replication count.
I ran the tests locally, including the new files, which passed.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-19954:
---
Attachment: 19954.v3.txt
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter
[
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369633#comment-16369633
]
Hudson commented on HBASE-19950:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4614 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4614/])
HBASE-19950 Introduce a ColumnValueFilter (chia7712: rev
a9a6eed372fa11dd2851635ac2c4bb6e1ca9dba7)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java
* (add)
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ColumnValueFilter.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterSerialization.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
* (edit) hbase-protocol/src/main/protobuf/Filter.proto
* (edit) hbase-protocol-shaded/src/main/protobuf/Filter.proto
> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
> Issue Type: New Feature
> Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-19950.master.001.patch,
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch,
> HBASE-19950.master.004.patch, HBASE-19950.master.005.patch,
> HBASE-19950.master.006.patch, HBASE-19950.master.007.patch,
> HBASE-19950.master.008.patch, HBASE-19950.master.009.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when
> specified condition is matched, this new filter will return the matched cell
> only. There're already some discussions in HBASE-19824.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19978) The keepalive logic is incomplete in ProcedureExecutor
[ https://issues.apache.org/jira/browse/HBASE-19978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19978: -- Resolution: Fixed Release Note: Completes keep-alive logic and then enables it; ProcedureExecutor Workers will spin up more threads when need settling back to the core count after the burst in demand has passed. Default keep-alive is one minute. Default core-count is CPUs/4 or 16, which ever is greater. Maximum is an arbitrary core-count * 10 (a limit that should never be hit and if it is, there is something else very wrong). Status: Resolved (was: Patch Available) Tried the patch. Watched the nice new test in operation watching spin up in threads as demand and then PE workers settling back to the core count as the test winds down. I pushed to see how it does with our last few flakies. One change is that I set the max at 10* instead of 4* core-count. Pushed to branch-2 and master. Thanks for nice work [~Apache9] where you are trying to make it so we can ship a hbase2 sooner rather than later sir. > The keepalive logic is incomplete in ProcedureExecutor > -- > > Key: HBASE-19978 > URL: https://issues.apache.org/jira/browse/HBASE-19978 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19978-v1.patch, HBASE-19978-v2.patch, > HBASE-19978.patch > > > The worker thread will just exit after keep alive time, and we never add it > back. The only way to add it back is through the stuck checker, this is not > correct. Here we should start new worker thread if it is under the core pool > size and there are pending procedures. > For now the default keep alive time is Long.MAX_VALUE which means no timeout > so no problem, but we do allow users to set it so we need to fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19147) All branch-2 unit tests pass
[ https://issues.apache.org/jira/browse/HBASE-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369620#comment-16369620 ] stack commented on HBASE-19147: --- bq. Do we need to go over the tests again to see the ignore ones? We could. I made one pass flagging tests that no longer made sense in AMv2 context and fixing others. Could do another pass but I think we in pretty good shape after we nail the current failing set (almost there). > All branch-2 unit tests pass > > > Key: HBASE-19147 > URL: https://issues.apache.org/jira/browse/HBASE-19147 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Priority: Blocker > Fix For: 2.0.0-beta-2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369617#comment-16369617
]
stack commented on HBASE-19954:
---
What about this question: "Why is the exception seen?/Why in hadoop3 do we get
this exception and not in hadoop2?"
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369613#comment-16369613
]
stack commented on HBASE-19954:
---
bq. Testing block reorder is orthogonal to shutdown hook installation.
No it is not. Both are meddling with HDFS in unorthodox ways.
FS should be same for all three tests, not change dependent on random order in
which tests run.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369611#comment-16369611
]
Ted Yu edited comment on HBASE-19954 at 2/20/18 12:23 AM:
--
Patch v2 adds audience annotation for the ShutdownHookManager class.
hasShutdownHook() is exercised by the TestBlockReorder against hadoop3.
If specific scenario is needed to test hasShutdownHook(), let me know.
was (Author: yuzhih...@gmail.com):
Patch v2 adds audience annotation for the ShutdownHookManager class.
hasShutdownHook() is exercised by the TestBlockReorder against hadoop3.
If specific scenario is desired to test hasShutdownHook(), let me know.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369611#comment-16369611
]
Ted Yu commented on HBASE-19954:
Patch v2 adds audience annotation for the ShutdownHookManager class.
hasShutdownHook() is exercised by the TestBlockReorder against hadoop3.
If specific scenario is desired to test hasShutdownHook(), let me know.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-19954:
---
Attachment: 19954.v2.txt
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369609#comment-16369609
]
Ted Yu commented on HBASE-19954:
Testing block reorder is orthogonal to shutdown hook installation.
When code for subtests (not just TestBlockReorder) is put into one subtest, it
deviates from the original intent of the tests.
ShutdownHook.install() is static. So more refactoring is needed for unit tests
to override.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19950) Introduce a ColumnValueFilter
[
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19950:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Push to master and branch-2. Thanks [~reidchan]
> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
> Issue Type: New Feature
> Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-19950.master.001.patch,
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch,
> HBASE-19950.master.004.patch, HBASE-19950.master.005.patch,
> HBASE-19950.master.006.patch, HBASE-19950.master.007.patch,
> HBASE-19950.master.008.patch, HBASE-19950.master.009.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when
> specified condition is matched, this new filter will return the matched cell
> only. There're already some discussions in HBASE-19824.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20019) Document the ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369512#comment-16369512 ] Chia-Ping Tsai commented on HBASE-20019: FYI [~reidchan] > Document the ColumnValueFilter > -- > > Key: HBASE-20019 > URL: https://issues.apache.org/jira/browse/HBASE-20019 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20019) Document the ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-20019: --- Fix Version/s: 2.0.0 > Document the ColumnValueFilter > -- > > Key: HBASE-20019 > URL: https://issues.apache.org/jira/browse/HBASE-20019 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20019) Document the ColumnValueFilter
Chia-Ping Tsai created HBASE-20019: -- Summary: Document the ColumnValueFilter Key: HBASE-20019 URL: https://issues.apache.org/jira/browse/HBASE-20019 Project: HBase Issue Type: Sub-task Reporter: Chia-Ping Tsai -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19950) Introduce a ColumnValueFilter
[
https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19950:
---
Issue Type: New Feature (was: Improvement)
> Introduce a ColumnValueFilter
> -
>
> Key: HBASE-19950
> URL: https://issues.apache.org/jira/browse/HBASE-19950
> Project: HBase
> Issue Type: New Feature
> Components: Filters
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-19950.master.001.patch,
> HBASE-19950.master.002.patch, HBASE-19950.master.003.patch,
> HBASE-19950.master.004.patch, HBASE-19950.master.005.patch,
> HBASE-19950.master.006.patch, HBASE-19950.master.007.patch,
> HBASE-19950.master.008.patch, HBASE-19950.master.009.patch
>
>
> Different from {{SingleColumnValueFilter}} which returns an entire row when
> specified condition is matched, this new filter will return the matched cell
> only. There're already some discussions in HBASE-19824.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369452#comment-16369452 ] Chia-Ping Tsai commented on HBASE-19970: Seem this patch is also in branch-1.4? > Remove unused functions from TableAuthManager > - > > Key: HBASE-19970 > URL: https://issues.apache.org/jira/browse/HBASE-19970 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0, 1.5.0 > > Attachments: HBASE-19970.master.001.patch > > > Functions deleted in TableAuthManager: > - setTableUserPermissions > - setTableGroupPermissions > - setNamespaceUserPermissions > - setNamespaceGroupPermissions > - writeTableToZooKeeper > - writeNamespaceToZooKeeper > To make sure it was not a bug, and that relevant functionality moved to some > alternate code path, tried to find out why and when these functions went out > of use. But just couldn't figure out...until i reached the patch which added > them. Looks like they were dead functions to start with :) > Jira which added them: HBASE-8409. Commit id: > ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.02.patch, HBASE-18294.03.patch, > HBASE-18294.04.patch, HBASE-18294.05.patch, HBASE-18294.06.patch, > HBASE-18294.07.patch, HBASE-18294.07.patch, HBASE-18294.08.patch, > HBASE-18294.09.patch, HBASE-18294.10.patch, HBASE-18294.11.patch, > HBASE-18294.11.patch, HBASE-18294.12.patch, HBASE-18294.13.patch, > HBASE-18294.15.patch, HBASE-18294.16.patch, HBASE-18294.branch-2.01.patch, > HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20003) WALLess HBase on Persistent Memory
[ https://issues.apache.org/jira/browse/HBASE-20003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368866#comment-16368866 ] ramkrishna.s.vasudevan commented on HBASE-20003: Thanks for the discussion and points [~apurtell]. Other than the above important point that Anoop mentioned here about how the WAL replay would get back a data which was actually not sent back to the client because the RS went down, another important aspect in this replica pipeline is that - We define the pipeline which is fixed like primary -> secondary-> tertiary. (as how the read replicas are available now). So any mutation goes through this pipeline. Pipeline is not randomly generated. The case of having 3 replicas and either 1 or 2 replicas are brought back, I think it falls under the following categories -> The mutation was not applied to the 2 replicas (and only primary is alive). The mutation itself would be a failure. Because we apply in the reverse order. If replica is a failure we won't write to primary also -> The mutation got applied to the 2 replicas but primary went down. So it falls under the current day WAL scenario that Anoop mentioned. -> The mutation got applied to the 3rd replica but 2nd replica went down. So still it means the mutation is a failure so retry that. But here again if the primary is down before retrying - still we bring back 3rd as new primary (switch over) which is actually alive at that time so it still falls back to the current day WAL case because we are actually not sure if the mutation was successful or not. So it will get back the latest data. But yes we need to have some remote pmem accessing also if the servers are actually reachable. > WALLess HBase on Persistent Memory > -- > > Key: HBASE-20003 > URL: https://issues.apache.org/jira/browse/HBASE-20003 > Project: HBase > Issue Type: New Feature >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > > This JIRA aims to make use of persistent memory (pmem) technologies in HBase. > One such usage is to make the Memstore to reside on pmem. Making a persistent > memstore would remove the need for WAL and paves way for a WALLess HBase. > The existing region replica feature could be used here and ensure the data > written to memstores are synchronously replicated to the replicas and ensure > strong consistency of the data. (pipeline model) > Advantages : > -Data Availability : Since the data across replicas are consistent > (synchronously written) our data is always 100 % available. > -Lower MTTR : It becomes easier/faster to switch over to the replicas on a > primary region failure as there is no WAL replay involved. Building the > memstore map data also is much faster than reading the WAL and replaying the > WAL. > -Possibility of bigger memstores : These pmems are designed to have more > memory than DRAMs so it would also enable us to have bigger sized memstores > which leads to lesser flushes/compaction IO. > -Removes the dependency of HDFS on the write path > Initial PoC has been designed and developed. Testing is underway and we would > publish the PoC results along with the design doc sooner. The PoC doc will > talk about the design decisions, the libraries considered to work with these > pmem devices, pros and cons of those libraries and the performance results. > Note : Next gen memory technologies using 3DXPoint gives persistent memory > feature. Such memory DIMMs are soon to appear in the market. The PoC is done > around Intel's ApachePass (AEP) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
