[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661663#comment-16661663 ] Duo Zhang commented on HBASE-21254: --- Can do this later. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661653#comment-16661653 ] Allan Yang commented on HBASE-21254: But we can use public Entry tryLockEntry(long id, long time) instead, we can still block the forceUpdateExecutor thread if some procedure is stuck. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661644#comment-16661644 ] Duo Zhang commented on HBASE-21254: --- But we do need to review the implementation again. As I found that, the actual deletion for a root procedure is done later in CompletedProcedureCleaner, so a successful procedure could also block us from removing a file... > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661643#comment-16661643 ] Duo Zhang commented on HBASE-21254: --- No it will be executed in a separated thread. In the log rolling thread we will just schedule a task into the forceUpdateExecutor. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661642#comment-16661642
]
Allan Yang commented on HBASE-21254:
In forceUpdateProcedure, we will acquire the execution lock before force update:
{code}
private void forceUpdateProcedure(long procId) throws IOException {
IdLock.Entry lockEntry = procExecutionLock.getLockEntry(procId);
try {
{code}
We will wait forever here if the procedure is stuck. And IIRC,
forceUpdateProcedure will be executed in the roll procedure log thread, which
will stuck it too.
> Need to find a way to limit the number of proc wal files
>
>
> Key: HBASE-21254
> URL: https://issues.apache.org/jira/browse/HBASE-21254
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch,
> HBASE-21254-v3.patch, HBASE-21254.patch
>
>
> For regionserver, we have a max wal file limitation, if we reach the
> limitation, we will trigger a flush on specific regions so that we can delete
> old wal files. But for proc wals, we do not have this mechanism, and it will
> be worse after HBASE-21233, as if there is an old procedure which can not
> make progress and do not persist its state, we need to keep the old proc wal
> file for ever...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661628#comment-16661628 ] Duo Zhang commented on HBASE-21254: --- Which lock? I do not get your point, about the 'wait for the lock when rolling log'... > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661625#comment-16661625 ] Allan Yang commented on HBASE-21254: I have a question for this one, what if a procedure is stuck there, and can't get the lock for it, will it stuck to wait for the lock when rolling log? > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647882#comment-16647882
]
Hudson commented on HBASE-21254:
Results for branch master
[build #541 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/541/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/541//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/541//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/541//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Need to find a way to limit the number of proc wal files
>
>
> Key: HBASE-21254
> URL: https://issues.apache.org/jira/browse/HBASE-21254
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch,
> HBASE-21254-v3.patch, HBASE-21254.patch
>
>
> For regionserver, we have a max wal file limitation, if we reach the
> limitation, we will trigger a flush on specific regions so that we can delete
> old wal files. But for proc wals, we do not have this mechanism, and it will
> be worse after HBASE-21233, as if there is an old procedure which can not
> make progress and do not persist its state, we need to keep the old proc wal
> file for ever...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647721#comment-16647721
]
Hudson commented on HBASE-21254:
Results for branch branch-2.1
[build #454 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/454/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/454//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/454//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/454//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Need to find a way to limit the number of proc wal files
>
>
> Key: HBASE-21254
> URL: https://issues.apache.org/jira/browse/HBASE-21254
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch,
> HBASE-21254-v3.patch, HBASE-21254.patch
>
>
> For regionserver, we have a max wal file limitation, if we reach the
> limitation, we will trigger a flush on specific regions so that we can delete
> old wal files. But for proc wals, we do not have this mechanism, and it will
> be worse after HBASE-21233, as if there is an old procedure which can not
> make progress and do not persist its state, we need to keep the old proc wal
> file for ever...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647715#comment-16647715
]
Hudson commented on HBASE-21254:
Results for branch branch-2.0
[build #939 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/939/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/939//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/939//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/939//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Need to find a way to limit the number of proc wal files
>
>
> Key: HBASE-21254
> URL: https://issues.apache.org/jira/browse/HBASE-21254
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch,
> HBASE-21254-v3.patch, HBASE-21254.patch
>
>
> For regionserver, we have a max wal file limitation, if we reach the
> limitation, we will trigger a flush on specific regions so that we can delete
> old wal files. But for proc wals, we do not have this mechanism, and it will
> be worse after HBASE-21233, as if there is an old procedure which can not
> make progress and do not persist its state, we need to keep the old proc wal
> file for ever...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647685#comment-16647685
]
Hudson commented on HBASE-21254:
Results for branch branch-2
[build #1378 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1378/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1378//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1378//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1378//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Need to find a way to limit the number of proc wal files
>
>
> Key: HBASE-21254
> URL: https://issues.apache.org/jira/browse/HBASE-21254
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch,
> HBASE-21254-v3.patch, HBASE-21254.patch
>
>
> For regionserver, we have a max wal file limitation, if we reach the
> limitation, we will trigger a flush on specific regions so that we can delete
> old wal files. But for proc wals, we do not have this mechanism, and it will
> be worse after HBASE-21233, as if there is an old procedure which can not
> make progress and do not persist its state, we need to keep the old proc wal
> file for ever...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647334#comment-16647334 ] stack commented on HBASE-21254: --- +1 to commit to branch-2.0 > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647269#comment-16647269 ] Duo Zhang commented on HBASE-21254: --- So let's commit? > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647192#comment-16647192 ] stack commented on HBASE-21254: --- Marked this critical. All goes to hell if we have a stuck procedure holding up WAL cleanup. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647189#comment-16647189 ] stack commented on HBASE-21254: --- I tried this on my test cluster and it seems to be working... 2018-10-11 16:29:35,912 WARN org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure WALs count=12 above the warning threshold 10. check running procedures to see if something is stuck. 2018-10-11 16:29:35,912 INFO org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=12 > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646473#comment-16646473
]
Hadoop QA commented on HBASE-21254:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} hbase-procedure: The patch generated 0 new + 20
unchanged - 1 fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} The patch passed checkstyle in hbase-server {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 33s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
28s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 43s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.token.TestZKSecretWatcher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21254 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943383/HBASE-21254-v3.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1cadf02d35d9 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646068#comment-16646068 ] Duo Zhang commented on HBASE-21254: --- Add more comments. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646043#comment-16646043
]
Duo Zhang commented on HBASE-21254:
---
{quote}
Does it log loudly enough when it is forcing rewrite of a stuck WAL edit?
{quote}
There are debug logs on every store.update when force updating a procedure.
> Need to find a way to limit the number of proc wal files
>
>
> Key: HBASE-21254
> URL: https://issues.apache.org/jira/browse/HBASE-21254
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch,
> HBASE-21254.patch
>
>
> For regionserver, we have a max wal file limitation, if we reach the
> limitation, we will trigger a flush on specific regions so that we can delete
> old wal files. But for proc wals, we do not have this mechanism, and it will
> be worse after HBASE-21233, as if there is an old procedure which can not
> make progress and do not persist its state, we need to keep the old proc wal
> file for ever...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645953#comment-16645953 ] stack commented on HBASE-21254: --- bq. stack I've replied on RB, so any other concerns? I think this is an important improvement. I responded. Please make new patch with the nice explanations you made for me folded into the patch and commit that. +1. I'll try testing it post commit (Does it log loudly enough when it is forcing rewrite of a stuck WAL edit? That'd be handy). Thanks. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645906#comment-16645906 ] Duo Zhang commented on HBASE-21254: --- [~stack] I've replied on RB, so any other concerns? I think this is an important improvement. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644510#comment-16644510
]
Hadoop QA commented on HBASE-21254:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} hbase-procedure: The patch generated 0 new + 20
unchanged - 1 fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} The patch passed checkstyle in hbase-server {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 57s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
1s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}203m
30s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}257m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21254 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943155/HBASE-21254-v2.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4a2ca0d4c0d0 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643588#comment-16643588
]
Hadoop QA commented on HBASE-21254:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
19s{color} | {color:red} hbase-procedure in master failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hbase-procedure: The patch generated 1 new + 20
unchanged - 1 fixed = 21 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 9s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
9s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 51s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21254 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943003/HBASE-21254-v1.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux dbbe9db80b44 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643110#comment-16643110 ] Duo Zhang commented on HBASE-21254: --- Ping [~stack] and [~allan163] for reviewing. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643109#comment-16643109 ] Duo Zhang commented on HBASE-21254: --- Review board link: https://reviews.apache.org/r/68963/ > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[
https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641698#comment-16641698
]
Hadoop QA commented on HBASE-21254:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
36s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s{color}
| {color:red} hbase-procedure generated 1 new + 9 unchanged - 0 fixed = 10
total (was 9) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} hbase-procedure: The patch generated 0 new + 19
unchanged - 1 fixed = 19 total (was 20) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} The patch passed checkstyle in hbase-server {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 8s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
14s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 6s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21254 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942793/HBASE-21254.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 67d2fee98c25 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633250#comment-16633250 ] Duo Zhang commented on HBASE-21254: --- I think maybe we could introduce something like the force flush at RS side. There is a holdingCleanupTracker for the oldest procedure wal file, if there are still procedure alive and only have record in this file then we can not deleted it, and we can get the procedure id through the holdingCleanupTracker. So maybe we can do this, add a listener call to indicate the upper layer, that these procedures need to be persist again, then the upper layer will call ProcedureStore.update to persist these procedures to the newest procedure wal file, then we are safe to delete the oldest files. I do not think this will break any assumptions? What do you guys think? [~stack] [~allan163]. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
