[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[ https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496128#comment-16496128 ] chenxu commented on HBASE-20507: nice FIX > Do not need to call recoverLease on the broken file when we fail to create a > wal writer > --- > > Key: HBASE-20507 > URL: https://issues.apache.org/jira/browse/HBASE-20507 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: 20507.addendum.patch, HBASE-20507.patch > > > I tried locally with a UT, if we overwrite a file which is currently being > written, the old file will be completed and then deleted. If you call close > on the previous file, a no lease exception will be thrown which means that > the file has already been completed. > So we do not need to close a file if it will be overwritten immediately, > since recoverLease may take a very long time... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463846#comment-16463846
]
Hudson commented on HBASE-20507:
Results for branch master
[build #320 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/320/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/320//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/320//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/320//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: 20507.addendum.patch, HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463373#comment-16463373
]
Hudson commented on HBASE-20507:
Results for branch branch-2.0
[build #256 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/256/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/256//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/256//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/256//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: 20507.addendum.patch, HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463320#comment-16463320
]
Hudson commented on HBASE-20507:
Results for branch branch-2
[build #692 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/692/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/692//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/692//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/692//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: 20507.addendum.patch, HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[ https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463232#comment-16463232 ] Duo Zhang commented on HBASE-20507: --- +1. > Do not need to call recoverLease on the broken file when we fail to create a > wal writer > --- > > Key: HBASE-20507 > URL: https://issues.apache.org/jira/browse/HBASE-20507 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: 20507.addendum.patch, HBASE-20507.patch > > > I tried locally with a UT, if we overwrite a file which is currently being > written, the old file will be completed and then deleted. If you call close > on the previous file, a no lease exception will be thrown which means that > the file has already been completed. > So we do not need to close a file if it will be overwritten immediately, > since recoverLease may take a very long time... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462961#comment-16462961
]
stack commented on HBASE-20507:
---
I pushed addendum (but did not call it out as so in the commit message).
[~Apache9] Let me know if I have this wrong.
{code}
commit 09ca6193890e8caffeb2951340b6a6fc10b5cfa0 (HEAD -> m, origin/master,
origin/HEAD)
Author: Michael Stack
Date: Thu May 3 12:26:12 2018 -0700
HBASE-20507 Do not need to call recoverLease on the broken file when we
fail to create a wal writer
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestOverwriteFileUnderConstruction.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestOverwriteFileUnderConstruction.java
index 7f75fdb268..aaeed1523f 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestOverwriteFileUnderConstruction.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestOverwriteFileUnderConstruction.java
@@ -22,6 +22,7 @@ import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
+import java.io.FileNotFoundException;
import java.io.IOException;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
@@ -95,6 +96,8 @@ public class TestOverwriteFileUnderConstruction {
out1.close();
// a successful close is also OK for us so no assertion here, we just
need to confirm that the
// data in the file are correct.
+} catch (FileNotFoundException fnfe) {
+ // hadoop3 throws one of these.
} catch (RemoteException e) {
// expected
assertThat(e.unwrapRemoteException(),
instanceOf(LeaseExpiredException.class));
{code}
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0, 2.0.1
>
> Attachments: 20507.addendum.patch, HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462639#comment-16462639
]
Ted Yu commented on HBASE-20507:
>From
>https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop3)/
{code}
[ERROR]
org.apache.hadoop.hbase.io.asyncfs.TestOverwriteFileUnderConstruction.testOverwrite(org.apache.hadoop.hbase.io.asyncfs.TestOverwriteFileUnderConstruction)
[ERROR] Run 1: TestOverwriteFileUnderConstruction.testOverwrite »
FileNotFound File does not ...
[ERROR] Run 2: TestOverwriteFileUnderConstruction.testOverwrite »
FileNotFound File does not ...
[ERROR] Run 3: TestOverwriteFileUnderConstruction.testOverwrite »
FileNotFound File does not ...
{code}
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460946#comment-16460946
]
Hudson commented on HBASE-20507:
Results for branch master
[build #318 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/318/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460887#comment-16460887
]
Hudson commented on HBASE-20507:
Results for branch branch-2
[build #685 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/685/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/685//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/685//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/685//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460868#comment-16460868
]
Hudson commented on HBASE-20507:
Results for branch branch-2.0
[build #249 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[ https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460468#comment-16460468 ] stack commented on HBASE-20507: --- Let me write your comment into code to avoid others asking why it is this way Will do it over on HBASE-20503... Thanks [~Apache9] > Do not need to call recoverLease on the broken file when we fail to create a > wal writer > --- > > Key: HBASE-20507 > URL: https://issues.apache.org/jira/browse/HBASE-20507 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.1.0, 2.0.1 > > Attachments: HBASE-20507.patch > > > I tried locally with a UT, if we overwrite a file which is currently being > written, the old file will be completed and then deleted. If you call close > on the previous file, a no lease exception will be thrown which means that > the file has already been completed. > So we do not need to close a file if it will be overwritten immediately, > since recoverLease may take a very long time... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460392#comment-16460392
]
Duo Zhang commented on HBASE-20507:
---
{quote}
Do we need to recover the lease in recoverAndClose?
{quote}
If we fail to write to DN, the standard way to recover this, is to bump the GS
at NN side, and reconnect to DN, finish the block, and then complete the file
normally. We can not call complete file directly, it does not work...
Here we use recoverLease to get the same result, and NN will help finishing the
block, since for wal we will not write to the stream any more.
And the recoverAndClose will be called in a thread pool in background I think?
It should not block the write request.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459689#comment-16459689
]
stack commented on HBASE-20507:
---
So, IIUC, this is different to what I see over in HBASE-20503 where I am
writing a WAL and then it breaks ("Stream already broken"). In this case we
roll the WAL and try to recover the file we had been writing too -- the one
that was reporting "stream already broken" -- in the stream's recoverAndClose.
Do we need to recover the lease in recoverAndClose?
On this patch, LGTM. Nice test. On the javac warning, I don't know how to
avoid. Perhaps someone else does. Meantime push it I'd say.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[ https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459603#comment-16459603 ] Duo Zhang commented on HBASE-20507: --- The javac warning is intentional, see the comments in the UT. Is there a way to ignore the check? > Do not need to call recoverLease on the broken file when we fail to create a > wal writer > --- > > Key: HBASE-20507 > URL: https://issues.apache.org/jira/browse/HBASE-20507 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20507.patch > > > I tried locally with a UT, if we overwrite a file which is currently being > written, the old file will be completed and then deleted. If you call close > on the previous file, a no lease exception will be thrown which means that > the file has already been completed. > So we do not need to close a file if it will be overwritten immediately, > since recoverLease may take a very long time... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[ https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459597#comment-16459597 ] Duo Zhang commented on HBASE-20507: --- When rolling a wal, if we fail to create an AsyncFSOutput, for now we will call recoverLease to close the file first, and then set overwrite to true to overwrite the old broken file and create a new one. The logic is inside the FanOutOneBlockAsyncDFSOutputHelper.createOutput, not the high level mechanism where we will open a new file to write if we fail to write to the current file. The UT here is used to confirm that we do not need to call recoverLease. If overwrite is true, NN will complete the pending file first and then delete it. > Do not need to call recoverLease on the broken file when we fail to create a > wal writer > --- > > Key: HBASE-20507 > URL: https://issues.apache.org/jira/browse/HBASE-20507 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20507.patch > > > I tried locally with a UT, if we overwrite a file which is currently being > written, the old file will be completed and then deleted. If you call close > on the previous file, a no lease exception will be thrown which means that > the file has already been completed. > So we do not need to close a file if it will be overwritten immediately, > since recoverLease may take a very long time... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[ https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458216#comment-16458216 ] stack commented on HBASE-20507: --- This looks great [~Apache9] Makes sense to me. What does "overwritten" mean in the above? We never overwrite a WAL? We roll and write the new? I am not following. Fix me up. Thanks. > Do not need to call recoverLease on the broken file when we fail to create a > wal writer > --- > > Key: HBASE-20507 > URL: https://issues.apache.org/jira/browse/HBASE-20507 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20507.patch > > > I tried locally with a UT, if we overwrite a file which is currently being > written, the old file will be completed and then deleted. If you call close > on the previous file, a no lease exception will be thrown which means that > the file has already been completed. > So we do not need to close a file if it will be overwritten immediately, > since recoverLease may take a very long time... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458044#comment-16458044
]
Hadoop QA commented on HBASE-20507:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 31s{color}
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 2s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}146m
37s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20507 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921177/HBASE-20507.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 31a34c846c97 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a136303833 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/12690/artifact/patchprocess/diff-compile-javac-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12690/testReport/ |
| Max. process+thread count | 4708 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[ https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457996#comment-16457996 ] Duo Zhang commented on HBASE-20507: --- [~stack] [~javaman_chen] FYI. > Do not need to call recoverLease on the broken file when we fail to create a > wal writer > --- > > Key: HBASE-20507 > URL: https://issues.apache.org/jira/browse/HBASE-20507 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20507.patch > > > I tried locally with a UT, if we overwrite a file which is currently being > written, the old file will be completed and then deleted. If you call close > on the previous file, a no lease exception will be thrown which means that > the file has already been completed. > So we do not need to close a file if it will be overwritten immediately, > since recoverLease may take a very long time... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
