[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[ https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542638#comment-16542638 ] Duo Zhang commented on HBASE-20686: --- Maybe we could make use of Proxy.newProxyInstance? The InvocationHandler does not need to know the number of parameters at compile time? Just say. > Asyncfs should retry upon RetryStartFileException > - > > Key: HBASE-20686 > URL: https://issues.apache.org/jira/browse/HBASE-20686 > Project: HBase > Issue Type: Bug > Components: asyncclient >Affects Versions: 2.0.0-beta-1 > Environment: HBase 2.0, Hadoop 3 with at-rest encryption >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20686.master.001.patch, > HBASE-20686.master.002.patch > > > In Hadoop-2.6 and above, HDFS client retries on RetryStartFileException when > NameNode experience encryption zone related issue. The code exists in > DFSOutputStream#newStreamForCreate(). (HDFS-6970) > In HBase-2's asyncfs implementation, > FanOutOneBlockAsyncDFSOutputHelper#createOutput() is somewhat an imitation of > HDFS's DFSOutputStream#newStreamForCreate(). However it does not retry upon > RetryStartFileException. So it is less resilient to such issues. > Also, DFSOutputStream#newStreamForCreate() upwraps RemoteExceptions, but > asyncfs does not. Therefore, hbase gets different exceptions than before. > File this jira to get this corrected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[
https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542633#comment-16542633
]
Wei-Chiu Chuang commented on HBASE-20686:
-
Thanks [~Apache9] yeah I've tried that but mocking a reflection based method
doesn't seem trivial. Ultimately you'd need to stub {{ClientProtocol.create()}}
but there are two versions of {{ClientProtocol.create()}} with different
parameter length depending on the Hadoop version, and {{Mockito.anyVararg()}}
doesn't work in this case. So the test code would need to somehow maintain
multiple versions of {{ClientProtocol.create()}} and it gets messier from
there.
> Asyncfs should retry upon RetryStartFileException
> -
>
> Key: HBASE-20686
> URL: https://issues.apache.org/jira/browse/HBASE-20686
> Project: HBase
> Issue Type: Bug
> Components: asyncclient
>Affects Versions: 2.0.0-beta-1
> Environment: HBase 2.0, Hadoop 3 with at-rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20686.master.001.patch,
> HBASE-20686.master.002.patch
>
>
> In Hadoop-2.6 and above, HDFS client retries on RetryStartFileException when
> NameNode experience encryption zone related issue. The code exists in
> DFSOutputStream#newStreamForCreate(). (HDFS-6970)
> In HBase-2's asyncfs implementation,
> FanOutOneBlockAsyncDFSOutputHelper#createOutput() is somewhat an imitation of
> HDFS's DFSOutputStream#newStreamForCreate(). However it does not retry upon
> RetryStartFileException. So it is less resilient to such issues.
> Also, DFSOutputStream#newStreamForCreate() upwraps RemoteExceptions, but
> asyncfs does not. Therefore, hbase gets different exceptions than before.
> File this jira to get this corrected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[
https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542627#comment-16542627
]
Hadoop QA commented on HBASE-20686:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
41s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
12s{color} | {color:red} hbase-server: The patch generated 4 new + 1 unchanged
- 0 fixed = 5 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
36s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 29s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}120m
49s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20686 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931428/HBASE-20686.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 89dc80bcb996 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ce82fd0f47 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13613/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13613/testReport/ |
| Max. process+thread count | 4805 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
http
[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[ https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542586#comment-16542586 ] Duo Zhang commented on HBASE-20686: --- I think you can use Mockito to create a mocked DistributedFileSystem? It is also not easy as we will get the ClientProtocol and call it directly, But I think this way is cleaner as the FILE_CREATOR is 'static final'... > Asyncfs should retry upon RetryStartFileException > - > > Key: HBASE-20686 > URL: https://issues.apache.org/jira/browse/HBASE-20686 > Project: HBase > Issue Type: Bug > Components: asyncclient >Affects Versions: 2.0.0-beta-1 > Environment: HBase 2.0, Hadoop 3 with at-rest encryption >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20686.master.001.patch, > HBASE-20686.master.002.patch > > > In Hadoop-2.6 and above, HDFS client retries on RetryStartFileException when > NameNode experience encryption zone related issue. The code exists in > DFSOutputStream#newStreamForCreate(). (HDFS-6970) > In HBase-2's asyncfs implementation, > FanOutOneBlockAsyncDFSOutputHelper#createOutput() is somewhat an imitation of > HDFS's DFSOutputStream#newStreamForCreate(). However it does not retry upon > RetryStartFileException. So it is less resilient to such issues. > Also, DFSOutputStream#newStreamForCreate() upwraps RemoteExceptions, but > asyncfs does not. Therefore, hbase gets different exceptions than before. > File this jira to get this corrected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[ https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542494#comment-16542494 ] Wei-Chiu Chuang commented on HBASE-20686: - Thanks [~chia7712] for review comments. I removed unwrapping other exceptions because they are not part of focus. Attach v002 patch. I realized the check for RetryStartFileException was in the code, but it was checked at the wrong place – RetryStartFileException is thrown from NameNode when a create() RPC is called. So removed the check of the exception that was previously there incorrectly. In addition, added a test to verify the check & retry is successful. The test code is very hacky because I had to replace a static final variable (FileCreator) in FanOutOneBlockAsyncDFSOutputHelper for test. The test code uses Java reflection to forcefully replace FileCreator. This approach was inspired by this web page: [https://caffinc.github.io/2015/12/static-final-java-junit-test/] [~Apache9] how do you feel about this test approach? I tried a few approaches but this one is the "cleanest" one. > Asyncfs should retry upon RetryStartFileException > - > > Key: HBASE-20686 > URL: https://issues.apache.org/jira/browse/HBASE-20686 > Project: HBase > Issue Type: Bug > Components: asyncclient >Affects Versions: 2.0.0-beta-1 > Environment: HBase 2.0, Hadoop 3 with at-rest encryption >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20686.master.001.patch, > HBASE-20686.master.002.patch > > > In Hadoop-2.6 and above, HDFS client retries on RetryStartFileException when > NameNode experience encryption zone related issue. The code exists in > DFSOutputStream#newStreamForCreate(). (HDFS-6970) > In HBase-2's asyncfs implementation, > FanOutOneBlockAsyncDFSOutputHelper#createOutput() is somewhat an imitation of > HDFS's DFSOutputStream#newStreamForCreate(). However it does not retry upon > RetryStartFileException. So it is less resilient to such issues. > Also, DFSOutputStream#newStreamForCreate() upwraps RemoteExceptions, but > asyncfs does not. Therefore, hbase gets different exceptions than before. > File this jira to get this corrected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[
https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504239#comment-16504239
]
Chia-Ping Tsai commented on HBASE-20686:
{quote}IMHO async client should be consistent with DFSOutputStream behavior.
I.e. if DFSOutputStream unwraps AccessControlException, async client should
unwrap it too.
{quote}
It is good if the consistent code can bring any benefit. Or add comment to
explain why we have to look up other exceptions.
> Asyncfs should retry upon RetryStartFileException
> -
>
> Key: HBASE-20686
> URL: https://issues.apache.org/jira/browse/HBASE-20686
> Project: HBase
> Issue Type: Bug
> Components: asyncclient
>Affects Versions: 2.0.0-beta-1
> Environment: HBase 2.0, Hadoop 3 with at-rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20686.master.001.patch
>
>
> In Hadoop-2.6 and above, HDFS client retries on RetryStartFileException when
> NameNode experience encryption zone related issue. The code exists in
> DFSOutputStream#newStreamForCreate(). (HDFS-6970)
> In HBase-2's asyncfs implementation,
> FanOutOneBlockAsyncDFSOutputHelper#createOutput() is somewhat an imitation of
> HDFS's DFSOutputStream#newStreamForCreate(). However it does not retry upon
> RetryStartFileException. So it is less resilient to such issues.
> Also, DFSOutputStream#newStreamForCreate() upwraps RemoteExceptions, but
> asyncfs does not. Therefore, hbase gets different exceptions than before.
> File this jira to get this corrected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[ https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504212#comment-16504212 ] Wei-Chiu Chuang commented on HBASE-20686: - Thanks for reviewing the patch, [~chia7712]. The jira summary was misleading. The patch unwraps these exceptions because IMHO async client should be consistent with DFSOutputStream behavior. I.e. if DFSOutputStream unwraps AccessControlException, async client should unwrap it too. If you think I should descope the patch and focus on handling RetryStartFileException, I can attach a patch as well. Thank you. > Asyncfs should retry upon RetryStartFileException > - > > Key: HBASE-20686 > URL: https://issues.apache.org/jira/browse/HBASE-20686 > Project: HBase > Issue Type: Bug > Components: asyncclient >Affects Versions: 2.0.0-beta-1 > Environment: HBase 2.0, Hadoop 3 with at-rest encryption >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20686.master.001.patch > > > In Hadoop-2.6 and above, HDFS client retries on RetryStartFileException when > NameNode experience encryption zone related issue. The code exists in > DFSOutputStream#newStreamForCreate(). (HDFS-6970) > In HBase-2's asyncfs implementation, > FanOutOneBlockAsyncDFSOutputHelper#createOutput() is somewhat an imitation of > HDFS's DFSOutputStream#newStreamForCreate(). However it does not retry upon > RetryStartFileException. So it is less resilient to such issues. > Also, DFSOutputStream#newStreamForCreate() upwraps RemoteExceptions, but > asyncfs does not. Therefore, hbase gets different exceptions than before. > File this jira to get this corrected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[
https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503293#comment-16503293
]
Chia-Ping Tsai commented on HBASE-20686:
{code:java}
+IOException e = re.unwrapRemoteException(
+AccessControlException.class,
+DSQuotaExceededException.class,
+QuotaByStorageTypeExceededException.class,
+FileAlreadyExistsException.class,
+FileNotFoundException.class,
+ParentNotDirectoryException.class,
+NSQuotaExceededException.class,
+RetryStartFileException.class,
+SafeModeException.class,
+UnresolvedPathException.class,
+SnapshotAccessControlException.class,
+UnknownCryptoProtocolVersionException.class);
+if (e instanceof RetryStartFileException) {
{code}
Why we try to look up other exception types if the exception we want to handle
is only the RetryStartFileException?
> Asyncfs should retry upon RetryStartFileException
> -
>
> Key: HBASE-20686
> URL: https://issues.apache.org/jira/browse/HBASE-20686
> Project: HBase
> Issue Type: Bug
> Components: asyncclient
>Affects Versions: 2.0.0-beta-1
> Environment: HBase 2.0, Hadoop 3 with at-rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20686.master.001.patch
>
>
> In Hadoop-2.6 and above, HDFS client retries on RetryStartFileException when
> NameNode experience encryption zone related issue. The code exists in
> DFSOutputStream#newStreamForCreate(). (HDFS-6970)
> In HBase-2's asyncfs implementation,
> FanOutOneBlockAsyncDFSOutputHelper#createOutput() is somewhat an imitation of
> HDFS's DFSOutputStream#newStreamForCreate(). However it does not retry upon
> RetryStartFileException. So it is less resilient to such issues.
> Also, DFSOutputStream#newStreamForCreate() upwraps RemoteExceptions, but
> asyncfs does not. Therefore, hbase gets different exceptions than before.
> File this jira to get this corrected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20686) Asyncfs should retry upon RetryStartFileException
[ https://issues.apache.org/jira/browse/HBASE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502079#comment-16502079 ] Wei-Chiu Chuang commented on HBASE-20686: - Attach a simple fix to make the behavior consistent with DFSOutputStream. [^HBASE-20686.master.001.patch] Will attach a test as well later. > Asyncfs should retry upon RetryStartFileException > - > > Key: HBASE-20686 > URL: https://issues.apache.org/jira/browse/HBASE-20686 > Project: HBase > Issue Type: Bug > Components: asyncclient >Affects Versions: 2.0.0-beta-1 > Environment: HBase 2.0, Hadoop 3 with at-rest encryption >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HBASE-20686.master.001.patch > > > In Hadoop-2.6 and above, HDFS client retries on RetryStartFileException when > NameNode experience encryption zone related issue. The code exists in > DFSOutputStream#newStreamForCreate(). (HDFS-6970) > In HBase-2's asyncfs implementation, > FanOutOneBlockAsyncDFSOutputHelper#createOutput() is somewhat an imitation of > HDFS's DFSOutputStream#newStreamForCreate(). However it does not retry upon > RetryStartFileException. So it is less resilient to such issues. > Also, DFSOutputStream#newStreamForCreate() upwraps RemoteExceptions, but > asyncfs does not. Therefore, hbase gets different exceptions than before. > File this jira to get this corrected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
