[jira] [Commented] (HDFS-15323) StandbyNode fails transition to active due to insufficient transaction tailing
[
https://issues.apache.org/jira/browse/HDFS-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097823#comment-17097823
]
Hadoop QA commented on HDFS-15323:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
54s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 31 unchanged - 0 fixed = 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 4s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.TestRefreshCallQueue |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29224/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15323 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13001860/HDFS-15323.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux cdf917faa221 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 82343790eeb |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| checkstyle |
[jira] [Commented] (HDFS-15323) StandbyNode fails transition to active due to insufficient transaction tailing
[
https://issues.apache.org/jira/browse/HDFS-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097821#comment-17097821
]
Chen Liang commented on HDFS-15323:
---
Thanks for the finding [~shv]! This is a tricky issue. In HDFS-14806, I
encountered a similar issue during boostrap standby, where one call limited by
QJM_RPC_MAX_TXNS is not sufficient to catch up. In HDFS-14806, the approach we
took is to just disable inprogress tailing during boostrap standby, and fall
back to the HTTP based edit tailing. The reasoning there was that inprogress
tailing was not meant to handle rare cases such as standup/failover and we can
avoid having multiple RPC calls. Do you think the same idea can apply here?
Apart from this, one minor comment. Can we add some logging around this logic?
So we can more easily identify issues like this in the future.
> StandbyNode fails transition to active due to insufficient transaction tailing
> --
>
> Key: HDFS-15323
> URL: https://issues.apache.org/jira/browse/HDFS-15323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.7
>Reporter: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15323.000.unitTest.patch, HDFS-15323.001.patch
>
>
> StandbyNode is asked to {{transitionToActive()}}. If it fell too far behind
> in tailing journal transaction (from QJM) it can crash with
> {{IllegalStateException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15310) RBF: Not proxy client's clientId and callId caused RetryCache invalid in NameNode.
[ https://issues.apache.org/jira/browse/HDFS-15310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-15310: --- Summary: RBF: Not proxy client's clientId and callId caused RetryCache invalid in NameNode. (was: RBF：Not proxy client's clientId and callId caused RetryCache invalid in NameNode.) > RBF: Not proxy client's clientId and callId caused RetryCache invalid in > NameNode. > -- > > Key: HDFS-15310 > URL: https://issues.apache.org/jira/browse/HDFS-15310 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: xuzq >Assignee: xuzq >Priority: Critical > > The RBF not proxy client's clientId and CallId to NameNode, it caused > RetryCache invalid in NameNode and some rpc may be failed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15078) RBF: Should check connection channel before sending RPC to namenode
[
https://issues.apache.org/jira/browse/HDFS-15078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15078:
---
Summary: RBF: Should check connection channel before sending RPC to
namenode (was: RBF: Should check connection channel before sending rpc to
namenode)
> RBF: Should check connection channel before sending RPC to namenode
> ---
>
> Key: HDFS-15078
> URL: https://issues.apache.org/jira/browse/HDFS-15078
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-15078.001.patch, HDFS-15078.002.patch
>
>
> dfsrouter logs show that
> {quote}
> 2019-12-20 04:11:26,724 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 6400 on , call org.apache.hadoop.hdfs.protocol.ClientProtocol.create from
> 10.83.164.11:56908 Call#2 Retry#0: output error
> 2019-12-20 04:11:26,724 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 125 on caught an exception
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
> at org.apache.hadoop.ipc.Server.channelWrite(Server.java:2731)
> at org.apache.hadoop.ipc.Server.access$2100(Server.java:134)
> at
> org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:1089)
> at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1161)
> at
> org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2109)
> at
> org.apache.hadoop.ipc.Server$Connection.access$400(Server.java:1229)
> at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:631)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2245)
> {quote}
> Maybe checking connection between client and router is better before
> sendingrpc to namenode
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15310) RBF：Not proxy client's clientId and callId caused RetryCache invalid in NameNode.
[ https://issues.apache.org/jira/browse/HDFS-15310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097810#comment-17097810 ] Íñigo Goiri commented on HDFS-15310: We've been dragging this for a while... We should reach some conclusion but this is a delicate topic. We can bring it up again in one of the lists. Anyway, HDFS-15079 covers the whole spectrum of solutions. > RBF：Not proxy client's clientId and callId caused RetryCache invalid in > NameNode. > - > > Key: HDFS-15310 > URL: https://issues.apache.org/jira/browse/HDFS-15310 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: xuzq >Assignee: xuzq >Priority: Critical > > The RBF not proxy client's clientId and CallId to NameNode, it caused > RetryCache invalid in NameNode and some rpc may be failed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15318) RBF: MountTable will not changed if only update the Order
[ https://issues.apache.org/jira/browse/HDFS-15318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-15318: --- Summary: RBF: MountTable will not changed if only update the Order (was: RBF：MountTable will not changed if only update the Order) > RBF: MountTable will not changed if only update the Order > - > > Key: HDFS-15318 > URL: https://issues.apache.org/jira/browse/HDFS-15318 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: xuzq >Priority: Major > > The MountTable will not changed, when we only change the Order. > The new order cannot work until Router has restarted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15312) Apply umask when creating directory by WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned HDFS-15312: -- Assignee: Ye Ni > Apply umask when creating directory by WebHDFS > -- > > Key: HDFS-15312 > URL: https://issues.apache.org/jira/browse/HDFS-15312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Reporter: Ye Ni >Assignee: Ye Ni >Priority: Minor > > WebHDFS methods for creating file/directories were always creating it with > 755 permissions as default for both files and directories. > The configured *fs.permissions.umask-mode* is intentionally ignored. > This Jira is to apply this setting in such scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15312) Apply umask when creating directory by WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097809#comment-17097809 ] Íñigo Goiri edited comment on HDFS-15312 at 5/2/20, 3:51 AM: - [~sodonnell], the proposal here is for explorer.js to read the config key and apply it when uploading the file. This is compatible with the discussion in HDFS-10488 which is to do it from the client side and not the server side. [~NickyYe], do you mind uploading the patch to make the different with HDFS-10488 clear? was (Author: elgoiri): [~sodonnell], the proposal here is for explorer.js to read the parameter and apply it when uploading the file. This is compatible with the discussion in HDFS-10488 which is to do it from the client side and not the server side. [~NickyYe], do you mind uploading the patch to make the different with HDFS-10488 clear? > Apply umask when creating directory by WebHDFS > -- > > Key: HDFS-15312 > URL: https://issues.apache.org/jira/browse/HDFS-15312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Reporter: Ye Ni >Priority: Minor > > WebHDFS methods for creating file/directories were always creating it with > 755 permissions as default for both files and directories. > The configured *fs.permissions.umask-mode* is intentionally ignored. > This Jira is to apply this setting in such scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15312) Apply umask when creating directory by WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097809#comment-17097809 ] Íñigo Goiri commented on HDFS-15312: [~sodonnell], the proposal here is for explorer.js to read the parameter and apply it when uploading the file. This is compatible with the discussion in HDFS-10488 which is to do it from the client side and not the server side. [~NickyYe], do you mind uploading the patch to make the different with HDFS-10488 clear? > Apply umask when creating directory by WebHDFS > -- > > Key: HDFS-15312 > URL: https://issues.apache.org/jira/browse/HDFS-15312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Reporter: Ye Ni >Priority: Minor > > WebHDFS methods for creating file/directories were always creating it with > 755 permissions as default for both files and directories. > The configured *fs.permissions.umask-mode* is intentionally ignored. > This Jira is to apply this setting in such scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15323) StandbyNode fails transition to active due to insufficient transaction tailing
[
https://issues.apache.org/jira/browse/HDFS-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-15323:
---
Status: Patch Available (was: Open)
Patch attached. Please review.
> StandbyNode fails transition to active due to insufficient transaction tailing
> --
>
> Key: HDFS-15323
> URL: https://issues.apache.org/jira/browse/HDFS-15323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.7
>Reporter: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15323.000.unitTest.patch, HDFS-15323.001.patch
>
>
> StandbyNode is asked to {{transitionToActive()}}. If it fell too far behind
> in tailing journal transaction (from QJM) it can crash with
> {{IllegalStateException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15323) StandbyNode fails transition to active due to insufficient transaction tailing
[
https://issues.apache.org/jira/browse/HDFS-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-15323:
---
Attachment: HDFS-15323.001.patch
> StandbyNode fails transition to active due to insufficient transaction tailing
> --
>
> Key: HDFS-15323
> URL: https://issues.apache.org/jira/browse/HDFS-15323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.7
>Reporter: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15323.000.unitTest.patch, HDFS-15323.001.patch
>
>
> StandbyNode is asked to {{transitionToActive()}}. If it fell too far behind
> in tailing journal transaction (from QJM) it can crash with
> {{IllegalStateException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097760#comment-17097760
]
Tsz-wo Sze commented on HDFS-15319:
---
The test failures seem related. When parent.isReference() == true, it probably
needs to be handled differently.
> Fix INode#isInLatestSnapshot() API
> --
>
> Key: HDFS-15319
> URL: https://issues.apache.org/jira/browse/HDFS-15319
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15319.000.patch
>
>
> isInLatestSnapshot() may return true in cases where an inode's ancesstors
> might not be in the latest snapshot.
> {code:java}
> // if parent is a reference node, parent must be a renamed node. We can
> // stop the check at the reference node.
> if (parent != null && parent.isReference()) {
> // TODO: Is it a bug to return true?
> // Some ancestor nodes may not be in the latest snapshot.
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14647) NPE during secure namenode startup
[
https://issues.apache.org/jira/browse/HDFS-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-14647:
---
Fix Version/s: 2.10.1
2.8.6
> NPE during secure namenode startup
> --
>
> Key: HDFS-14647
> URL: https://issues.apache.org/jira/browse/HDFS-14647
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Fix For: 3.3.0, 2.8.6, 3.1.4, 3.2.2, 2.10.1
>
> Attachments: HDFS-14647-2.002.patch, HDFS-14647-trunk.001.patch,
> HDFS-14647-trunk.002.patch, HDFS-14647-trunk.003.patch,
> HDFS-14647-trunk.004.patch, HDFS-14647.001.patch
>
>
> In secure HDFS, during Namenode loading fsimage, when hitting Namenode
> through the REST API, below exception would be thrown out. (This is in
> version 2.8.2)
> {quote}org.apache.hadoop.hdfs.web.resources.ExceptionHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getTokenUGI(JspHelper.java:283)
> at org.apache.hadoop.hdfs.server.common.JspHelper.getUGI(JspHelper.java:226)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:54)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:42)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:87)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1353)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
[jira] [Commented] (HDFS-14647) NPE during secure namenode startup
[
https://issues.apache.org/jira/browse/HDFS-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097748#comment-17097748
]
Konstantin Shvachko commented on HDFS-14647:
[~fengnanli] for future reference we should not skip branches: if you committed
to 2.8, you should have committed to 2.10 and 2.9.
Also worth reflecting in the "Fix Version/s:" field, otherwise we wont know.
> NPE during secure namenode startup
> --
>
> Key: HDFS-14647
> URL: https://issues.apache.org/jira/browse/HDFS-14647
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14647-2.002.patch, HDFS-14647-trunk.001.patch,
> HDFS-14647-trunk.002.patch, HDFS-14647-trunk.003.patch,
> HDFS-14647-trunk.004.patch, HDFS-14647.001.patch
>
>
> In secure HDFS, during Namenode loading fsimage, when hitting Namenode
> through the REST API, below exception would be thrown out. (This is in
> version 2.8.2)
> {quote}org.apache.hadoop.hdfs.web.resources.ExceptionHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getTokenUGI(JspHelper.java:283)
> at org.apache.hadoop.hdfs.server.common.JspHelper.getUGI(JspHelper.java:226)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:54)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:42)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:87)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1353)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
>
[jira] [Updated] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-15293: --- Attachment: HDFS-15323.000.unitTest.patch > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: multi-sbnn > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-15293: --- Attachment: (was: HDFS-15323.000.unitTest.patch) > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: multi-sbnn > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15323) StandbyNode fails transition to active due to insufficient transaction tailing
[
https://issues.apache.org/jira/browse/HDFS-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-15323:
---
Attachment: HDFS-15323.000.unitTest.patch
> StandbyNode fails transition to active due to insufficient transaction tailing
> --
>
> Key: HDFS-15323
> URL: https://issues.apache.org/jira/browse/HDFS-15323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.7
>Reporter: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15323.000.unitTest.patch
>
>
> StandbyNode is asked to {{transitionToActive()}}. If it fell too far behind
> in tailing journal transaction (from QJM) it can crash with
> {{IllegalStateException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15323) StandbyNode fails transition to active due to insufficient transaction tailing
[
https://issues.apache.org/jira/browse/HDFS-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097741#comment-17097741
]
Konstantin Shvachko commented on HDFS-15323:
Took some time to figure out the problem.
# The main problem is in {{EditLogTailer.catchupDuringFailover()}}, which is
called by {{startActiveServices()}} during failover.
{{catchupDuringFailover()}} runs {{doTailEdits()}} once and returns. While
{{doTailEdits()}} is supposed to bring one portion of edits and return. Getting
one portion of edits for {{doTailEdits()}} is fine for periodic tailing, since
on the next iteration it will get the next portion, and then the next.
But it is not enough for {{catchupDuringFailover()}}, because catching up
should fully read all transactions up to the last state of the previous Active
NN. But it reads only up to a fixed number of edits ({{QJM_RPC_MAX_TXNS}}). So
if StandbyNode falls far behind then it may not fully catch up with only one
portion of edits.
# StandbyNode can fall far behind during checkpoint process. It does
{{saveNamespace()}}, which hold the namesystem lock and during this time
{{EditLogTailer}} cannot apply transactions to the namespace. I compared
transaction ids on StandbyNode and Active when standby was creating a
checkpoint. Standby was ~1M transaction behind at some point, while between
checkpoints the lag is around 100 tx. So if you failover during this time, the
checkpoint thread is interrupted, image saving is cancelled, but catching up is
not done properly.
You need a fairly large image to hit this error.
We caught this problem in our 2.10 branch, but it exists in all HDFS versions.
Attaching a unit test to reproduce the problem.
> StandbyNode fails transition to active due to insufficient transaction tailing
> --
>
> Key: HDFS-15323
> URL: https://issues.apache.org/jira/browse/HDFS-15323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.7
>Reporter: Konstantin Shvachko
>Priority: Major
>
> StandbyNode is asked to {{transitionToActive()}}. If it fell too far behind
> in tailing journal transaction (from QJM) it can crash with
> {{IllegalStateException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15323) StandbyNode fails transition to active due to insufficient transaction tailing
[
https://issues.apache.org/jira/browse/HDFS-15323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097735#comment-17097735
]
Konstantin Shvachko commented on HDFS-15323:
This is the exception it throws:
{noformat:nowrap}
2020-04-30 02:07:19,087 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
Error encountered requiring NN shutdown. Shutting down immediately.
java.lang.IllegalStateException: Cannot start writing at txid 449161330348 when
there is a stream available for read:
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@68063923
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.openForWrite(FSEditLog.java:321)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:1148)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1768)
at
org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
at
org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:64)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:60)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1627)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:1513)
at
org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:112)
at
org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:5409)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:622)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1027)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2373)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2369)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2369)
2020-04-30 02:07:19,100 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2020-04-30 02:07:19,101 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down NameNode at
/
{noformat}
The exception means that SBN did not finish catching up to the final state of
the journal. There is more transactions there than it consumed while catching
up.
> StandbyNode fails transition to active due to insufficient transaction tailing
> --
>
> Key: HDFS-15323
> URL: https://issues.apache.org/jira/browse/HDFS-15323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.7
>Reporter: Konstantin Shvachko
>Priority: Major
>
> StandbyNode is asked to {{transitionToActive()}}. If it fell too far behind
> in tailing journal transaction (from QJM) it can crash with
> {{IllegalStateException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15323) StandbyNode fails transition to active due to insufficient transaction tailing
Konstantin Shvachko created HDFS-15323:
--
Summary: StandbyNode fails transition to active due to
insufficient transaction tailing
Key: HDFS-15323
URL: https://issues.apache.org/jira/browse/HDFS-15323
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode, qjm
Affects Versions: 2.7.7
Reporter: Konstantin Shvachko
StandbyNode is asked to {{transitionToActive()}}. If it fell too far behind in
tailing journal transaction (from QJM) it can crash with
{{IllegalStateException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15264) Backport HDFS-13571,HDFS-15149 to branch-3.1, branch-2.10
[ https://issues.apache.org/jira/browse/HDFS-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097632#comment-17097632 ] Ayush Saxena commented on HDFS-15264: - Thanx [~leosun08] for the patches, The branch-2 v05 Seems to have some findbugs warnings, Can you check. Can you help point the changes made with respect to trunk. Would be easy reviewing then. :) > Backport HDFS-13571,HDFS-15149 to branch-3.1, branch-2.10 > - > > Key: HDFS-15264 > URL: https://issues.apache.org/jira/browse/HDFS-15264 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-15264-branch-2.10.001.patch, > HDFS-15264-branch-2.10.002.patch, HDFS-15264-branch-2.10.003.patch, > HDFS-15264-branch-2.10.004.patch, HDFS-15264-branch-2.10.005.patch, > HDFS-15264-branch-3.1.001.patch, HDFS-15264-branch-3.1.002.patch, > HDFS-15264-branch-3.1.003.patch, HDFS-15264-branch-3.1.004.patch, > HDFS-15264.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14647) NPE during secure namenode startup
[
https://issues.apache.org/jira/browse/HDFS-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097626#comment-17097626
]
Fengnan Li commented on HDFS-14647:
---
[~vagarychen] Sorry for coming late. I only committed to 2.8, but it also
applies to 2.10.
> NPE during secure namenode startup
> --
>
> Key: HDFS-14647
> URL: https://issues.apache.org/jira/browse/HDFS-14647
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14647-2.002.patch, HDFS-14647-trunk.001.patch,
> HDFS-14647-trunk.002.patch, HDFS-14647-trunk.003.patch,
> HDFS-14647-trunk.004.patch, HDFS-14647.001.patch
>
>
> In secure HDFS, during Namenode loading fsimage, when hitting Namenode
> through the REST API, below exception would be thrown out. (This is in
> version 2.8.2)
> {quote}org.apache.hadoop.hdfs.web.resources.ExceptionHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getTokenUGI(JspHelper.java:283)
> at org.apache.hadoop.hdfs.server.common.JspHelper.getUGI(JspHelper.java:226)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:54)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:42)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:87)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1353)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at
[jira] [Commented] (HDFS-15316) Deletion failure should not remove directory from snapshottables
[
https://issues.apache.org/jira/browse/HDFS-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097616#comment-17097616
]
Hadoop QA commented on HDFS-15316:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
3s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 20s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.TestRefreshCallQueue |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29223/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15316 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13001836/HDFS-15316.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2267c4bce2cd 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 82343790eeb |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/29223/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/29223/testReport/ |
| Max.
[jira] [Commented] (HDFS-14647) NPE during secure namenode startup
[
https://issues.apache.org/jira/browse/HDFS-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097589#comment-17097589
]
Chen Liang commented on HDFS-14647:
---
Thanks Konstantin, I have committed the branch-2 patch to branch-2.10.
> NPE during secure namenode startup
> --
>
> Key: HDFS-14647
> URL: https://issues.apache.org/jira/browse/HDFS-14647
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14647-2.002.patch, HDFS-14647-trunk.001.patch,
> HDFS-14647-trunk.002.patch, HDFS-14647-trunk.003.patch,
> HDFS-14647-trunk.004.patch, HDFS-14647.001.patch
>
>
> In secure HDFS, during Namenode loading fsimage, when hitting Namenode
> through the REST API, below exception would be thrown out. (This is in
> version 2.8.2)
> {quote}org.apache.hadoop.hdfs.web.resources.ExceptionHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getTokenUGI(JspHelper.java:283)
> at org.apache.hadoop.hdfs.server.common.JspHelper.getUGI(JspHelper.java:226)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:54)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:42)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:87)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1353)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
>
[jira] [Commented] (HDFS-15305) Extend ViewFS and provide ViewFSOverloadScheme implementation with scheme configurable.
[ https://issues.apache.org/jira/browse/HDFS-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097572#comment-17097572 ] Uma Maheswara Rao G commented on HDFS-15305: Thank you [~virajith] for reviews. I have updated PR by addressing the comments. Please take a look. Thanks! > Extend ViewFS and provide ViewFSOverloadScheme implementation with scheme > configurable. > --- > > Key: HDFS-15305 > URL: https://issues.apache.org/jira/browse/HDFS-15305 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hadoop-client, hdfs-client, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > Provide ViewFsOverloadScheme implementation by extending ViewFileSystem class. > # When target scheme and uri scheme matches, it should handle to create > target filesystems different way than using FileSystem.get API. > # Provide the flexibility to configure overload scheme. > ex: by setting hdfs scheme and impl to ViewFsOverloadScheme, users should be > able to continue working with hdfs scheme uris and should be able to mount > any hadoop compatible file systems as target. It will follow the same mount > link configuration pattern as ViewFileSystem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14647) NPE during secure namenode startup
[
https://issues.apache.org/jira/browse/HDFS-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097546#comment-17097546
]
Konstantin Shvachko commented on HDFS-14647:
I think it makes sense to commit to branch 2.10. I have seen it on our
clusters. Looks scary and annoying.
> NPE during secure namenode startup
> --
>
> Key: HDFS-14647
> URL: https://issues.apache.org/jira/browse/HDFS-14647
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14647-2.002.patch, HDFS-14647-trunk.001.patch,
> HDFS-14647-trunk.002.patch, HDFS-14647-trunk.003.patch,
> HDFS-14647-trunk.004.patch, HDFS-14647.001.patch
>
>
> In secure HDFS, during Namenode loading fsimage, when hitting Namenode
> through the REST API, below exception would be thrown out. (This is in
> version 2.8.2)
> {quote}org.apache.hadoop.hdfs.web.resources.ExceptionHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.common.JspHelper.getTokenUGI(JspHelper.java:283)
> at org.apache.hadoop.hdfs.server.common.JspHelper.getUGI(JspHelper.java:226)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:54)
> at
> org.apache.hadoop.hdfs.web.resources.UserProvider.getValue(UserProvider.java:42)
> at
> com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:87)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1353)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>
[jira] [Commented] (HDFS-15316) Deletion failure should not remove directory from snapshottables
[
https://issues.apache.org/jira/browse/HDFS-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097474#comment-17097474
]
hemanthboyina commented on HDFS-15316:
--
thanks [~ayushtkn] for review
updated the patch , please review
{quote}Secondly, can you help when in actual scenario
{quote}
it happened in very rare scenario , though it can be a safeguard condition to
check
> Deletion failure should not remove directory from snapshottables
>
>
> Key: HDFS-15316
> URL: https://issues.apache.org/jira/browse/HDFS-15316
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-15316.001.patch, HDFS-15316.002.patch
>
>
> If deleting a directory doesn't succeeds , still we are removing directory
> from snapshottables
> this makes the system inconsistent , we will be able to create snapshots but
> snapshot diff throws Directory is not snaphottable
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15316) Deletion failure should not remove directory from snapshottables
[ https://issues.apache.org/jira/browse/HDFS-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-15316: - Attachment: HDFS-15316.002.patch > Deletion failure should not remove directory from snapshottables > > > Key: HDFS-15316 > URL: https://issues.apache.org/jira/browse/HDFS-15316 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15316.001.patch, HDFS-15316.002.patch > > > If deleting a directory doesn't succeeds , still we are removing directory > from snapshottables > this makes the system inconsistent , we will be able to create snapshots but > snapshot diff throws Directory is not snaphottable -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13571) Deadnode detection
[ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-13571: --- Issue Type: New Feature (was: Improvement) > Deadnode detection > -- > > Key: HDFS-13571 > URL: https://issues.apache.org/jira/browse/HDFS-13571 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Affects Versions: 2.4.0, 2.6.0, 3.0.2 >Reporter: Gang Xie >Assignee: Lisheng Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: DeadNodeDetectorDesign.pdf, HDFS-13571-2.6.diff, node > status machine.png > > > Currently, the information of the dead datanode in DFSInputStream in stored > locally. So, it could not be shared among the inputstreams of the same > DFSClient. In our production env, every days, some datanodes dies with > different causes. At this time, after the first inputstream blocked and > detect this, it could share this information to others in the same DFSClient, > thus, the ohter inputstreams are still blocked by the dead node for some > time, which could cause bad service latency. > To eliminate this impact from dead datanode, we designed a dead datanode > detector, which detect the dead ones in advance, and share this information > among all the inputstreams in the same client. This improvement has being > online for some months and works fine. So, we decide to port to the 3.0 (the > version used in our production env is 2.4 and 2.6). > I will do the porting work and upload the code later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13571) Deadnode detection
[ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097471#comment-17097471 ] Wei-Chiu Chuang commented on HDFS-13571: This looks like a new feature rather than an improvement. > Deadnode detection > -- > > Key: HDFS-13571 > URL: https://issues.apache.org/jira/browse/HDFS-13571 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.4.0, 2.6.0, 3.0.2 >Reporter: Gang Xie >Assignee: Lisheng Sun >Priority: Major > Fix For: 3.3.0 > > Attachments: DeadNodeDetectorDesign.pdf, HDFS-13571-2.6.diff, node > status machine.png > > > Currently, the information of the dead datanode in DFSInputStream in stored > locally. So, it could not be shared among the inputstreams of the same > DFSClient. In our production env, every days, some datanodes dies with > different causes. At this time, after the first inputstream blocked and > detect this, it could share this information to others in the same DFSClient, > thus, the ohter inputstreams are still blocked by the dead node for some > time, which could cause bad service latency. > To eliminate this impact from dead datanode, we designed a dead datanode > detector, which detect the dead ones in advance, and share this information > among all the inputstreams in the same client. This improvement has being > online for some months and works fine. So, we decide to port to the 3.0 (the > version used in our production env is 2.4 and 2.6). > I will do the porting work and upload the code later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15264) Backport HDFS-13571,HDFS-15149 to branch-3.1, branch-2.10
[
https://issues.apache.org/jira/browse/HDFS-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097459#comment-17097459
]
Hadoop QA commented on HDFS-15264:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
55s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.10 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
39s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} branch-2.10 passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} branch-2.10 passed with JDK Private
Build-1.8.0_252-8u252-b09-1~16.04-b09 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} branch-2.10 passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} branch-2.10 passed with JDK Private
Build-1.8.0_252-8u252-b09-1~16.04-b09 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
46s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
9s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in branch-2.10
has 1 extant findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
44s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 10
extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} the patch passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_252-8u252-b09-1~16.04-b09 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 151
unchanged - 3 fixed = 151 total (was 154) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_252-8u252-b09-1~16.04-b09 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
45s{color} | {color:green} the patch passed
[jira] [Commented] (HDFS-15314) Add concat fs command
[ https://issues.apache.org/jira/browse/HDFS-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097453#comment-17097453 ] Jinglun commented on HDFS-15314: Sorry I didn't know that. Thanks [~weichiu] your kindly reminding. > Add concat fs command > - > > Key: HDFS-15314 > URL: https://issues.apache.org/jira/browse/HDFS-15314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Minor > > We should add one concat fs command for ease of use. It concatenates existing > source files into the target file using FileSystem.concat(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15264) Backport HDFS-13571,HDFS-15149 to branch-3.1, branch-2.10
[ https://issues.apache.org/jira/browse/HDFS-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097366#comment-17097366 ] Lisheng Sun commented on HDFS-15264: the branch-2.10.005.patch fixed the checkstyle. the findbugs and failed ut should be this patch. [~ayushtkn] [~elgoiri] [~weichiu] Could you help review this patch? Thank you. > Backport HDFS-13571,HDFS-15149 to branch-3.1, branch-2.10 > - > > Key: HDFS-15264 > URL: https://issues.apache.org/jira/browse/HDFS-15264 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-15264-branch-2.10.001.patch, > HDFS-15264-branch-2.10.002.patch, HDFS-15264-branch-2.10.003.patch, > HDFS-15264-branch-2.10.004.patch, HDFS-15264-branch-2.10.005.patch, > HDFS-15264-branch-3.1.001.patch, HDFS-15264-branch-3.1.002.patch, > HDFS-15264-branch-3.1.003.patch, HDFS-15264-branch-3.1.004.patch, > HDFS-15264.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15264) Backport HDFS-13571,HDFS-15149 to branch-3.1, branch-2.10
[ https://issues.apache.org/jira/browse/HDFS-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15264: --- Attachment: HDFS-15264-branch-2.10.005.patch > Backport HDFS-13571,HDFS-15149 to branch-3.1, branch-2.10 > - > > Key: HDFS-15264 > URL: https://issues.apache.org/jira/browse/HDFS-15264 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-15264-branch-2.10.001.patch, > HDFS-15264-branch-2.10.002.patch, HDFS-15264-branch-2.10.003.patch, > HDFS-15264-branch-2.10.004.patch, HDFS-15264-branch-2.10.005.patch, > HDFS-15264-branch-3.1.001.patch, HDFS-15264-branch-3.1.002.patch, > HDFS-15264-branch-3.1.003.patch, HDFS-15264-branch-3.1.004.patch, > HDFS-15264.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097286#comment-17097286
]
Hadoop QA commented on HDFS-15319:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
59s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 43 unchanged - 1 fixed = 43 total (was 44) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 38s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.TestRefreshCallQueue |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29221/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15319 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13001796/HDFS-15319.000.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient
[jira] [Commented] (HDFS-15320) StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter
[
https://issues.apache.org/jira/browse/HDFS-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097276#comment-17097276
]
Akira Ajisaka commented on HDFS-15320:
--
Hi [~clayb] and [~aengineer], would you check the GitHub PR?
> StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter
> -
>
> Key: HDFS-15320
> URL: https://issues.apache.org/jira/browse/HDFS-15320
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Environment: HostRestrictingAuthorizationFilter (HDFS-14234) is
> enabled
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>
> When there is a request to "http://:/" without "webhdfs/v1"
> suffix, DN returns 500 response code and throws
> StringIndexOutOfBoundsException as follows:
> {noformat}
> 2020-05-01 16:10:20,220 ERROR
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler:
> Exception in HostRestrictingAuthorizationFilterHandler
> java.lang.StringIndexOutOfBoundsException: String index out of range: -10
> at java.base/java.lang.String.substring(String.java:1841)
> at
> org.apache.hadoop.hdfs.server.common.HostRestrictingAuthorizationFilter.handleInteraction(HostRestrictingAuthorizationFilter.java:234)
> at
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler.channelRead0(HostRestrictingAuthorizationFilterHandler.java:155)
> at
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler.channelRead0(HostRestrictingAuthorizationFilterHandler.java:58)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15320) StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter
[
https://issues.apache.org/jira/browse/HDFS-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-15320:
-
Status: Patch Available (was: In Progress)
> StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter
> -
>
> Key: HDFS-15320
> URL: https://issues.apache.org/jira/browse/HDFS-15320
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Environment: HostRestrictingAuthorizationFilter (HDFS-14234) is
> enabled
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>
> When there is a request to "http://:/" without "webhdfs/v1"
> suffix, DN returns 500 response code and throws
> StringIndexOutOfBoundsException as follows:
> {noformat}
> 2020-05-01 16:10:20,220 ERROR
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler:
> Exception in HostRestrictingAuthorizationFilterHandler
> java.lang.StringIndexOutOfBoundsException: String index out of range: -10
> at java.base/java.lang.String.substring(String.java:1841)
> at
> org.apache.hadoop.hdfs.server.common.HostRestrictingAuthorizationFilter.handleInteraction(HostRestrictingAuthorizationFilter.java:234)
> at
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler.channelRead0(HostRestrictingAuthorizationFilterHandler.java:155)
> at
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler.channelRead0(HostRestrictingAuthorizationFilterHandler.java:58)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15320) StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter
[
https://issues.apache.org/jira/browse/HDFS-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-15320 started by Akira Ajisaka.
> StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter
> -
>
> Key: HDFS-15320
> URL: https://issues.apache.org/jira/browse/HDFS-15320
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Environment: HostRestrictingAuthorizationFilter (HDFS-14234) is
> enabled
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>
> When there is a request to "http://:/" without "webhdfs/v1"
> suffix, DN returns 500 response code and throws
> StringIndexOutOfBoundsException as follows:
> {noformat}
> 2020-05-01 16:10:20,220 ERROR
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler:
> Exception in HostRestrictingAuthorizationFilterHandler
> java.lang.StringIndexOutOfBoundsException: String index out of range: -10
> at java.base/java.lang.String.substring(String.java:1841)
> at
> org.apache.hadoop.hdfs.server.common.HostRestrictingAuthorizationFilter.handleInteraction(HostRestrictingAuthorizationFilter.java:234)
> at
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler.channelRead0(HostRestrictingAuthorizationFilterHandler.java:155)
> at
> org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler.channelRead0(HostRestrictingAuthorizationFilterHandler.java:58)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15322) Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same.
[ https://issues.apache.org/jira/browse/HDFS-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-15322: -- Assignee: Uma Maheswara Rao G > Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris > schemes are same. > > > Key: HDFS-15322 > URL: https://issues.apache.org/jira/browse/HDFS-15322 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, nflyFs, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15322) Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same.
Uma Maheswara Rao G created HDFS-15322: -- Summary: Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Key: HDFS-15322 URL: https://issues.apache.org/jira/browse/HDFS-15322 Project: Hadoop HDFS Issue Type: Sub-task Components: viewfsOverloadScheme, nflyFs, fs, viewfs Affects Versions: 3.2.1 Reporter: Uma Maheswara Rao G -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15321) Make DFSAdmin tool to work with ViewFSOverloadScheme
Uma Maheswara Rao G created HDFS-15321: -- Summary: Make DFSAdmin tool to work with ViewFSOverloadScheme Key: HDFS-15321 URL: https://issues.apache.org/jira/browse/HDFS-15321 Project: Hadoop HDFS Issue Type: Sub-task Components: dfsadmin, fs, viewfs Affects Versions: 3.2.1 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15320) StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter
Akira Ajisaka created HDFS-15320:
Summary: StringIndexOutOfBoundsException in
HostRestrictingAuthorizationFilter
Key: HDFS-15320
URL: https://issues.apache.org/jira/browse/HDFS-15320
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Environment: HostRestrictingAuthorizationFilter (HDFS-14234) is enabled
Reporter: Akira Ajisaka
Assignee: Akira Ajisaka
When there is a request to "http://:/" without "webhdfs/v1"
suffix, DN returns 500 response code and throws StringIndexOutOfBoundsException
as follows:
{noformat}
2020-05-01 16:10:20,220 ERROR
org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler:
Exception in HostRestrictingAuthorizationFilterHandler
java.lang.StringIndexOutOfBoundsException: String index out of range: -10
at java.base/java.lang.String.substring(String.java:1841)
at
org.apache.hadoop.hdfs.server.common.HostRestrictingAuthorizationFilter.handleInteraction(HostRestrictingAuthorizationFilter.java:234)
at
org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler.channelRead0(HostRestrictingAuthorizationFilterHandler.java:155)
at
org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler.channelRead0(HostRestrictingAuthorizationFilterHandler.java:58)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15264) Backport HDFS-13571,HDFS-15149 to branch-3.1, branch-2.10
[
https://issues.apache.org/jira/browse/HDFS-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097212#comment-17097212
]
Hadoop QA commented on HDFS-15264:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.10 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} branch-2.10 passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} branch-2.10 passed with JDK Private
Build-1.8.0_252-8u252-b09-1~16.04-b09 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} branch-2.10 passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} branch-2.10 passed with JDK Private
Build-1.8.0_252-8u252-b09-1~16.04-b09 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
31s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
55s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in branch-2.10
has 1 extant findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
29s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 10
extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_252-8u252-b09-1~16.04-b09 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
151 unchanged - 3 fixed = 153 total (was 154) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_252-8u252-b09-1~16.04-b09 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
34s{color} | {color:green} the patch passed
[jira] [Updated] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15319:
---
Status: Patch Available (was: Open)
> Fix INode#isInLatestSnapshot() API
> --
>
> Key: HDFS-15319
> URL: https://issues.apache.org/jira/browse/HDFS-15319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15319.000.patch
>
>
> isInLatestSnapshot() may return true in cases where an inode's ancesstors
> might not be in the latest snapshot.
> {code:java}
> // if parent is a reference node, parent must be a renamed node. We can
> // stop the check at the reference node.
> if (parent != null && parent.isReference()) {
> // TODO: Is it a bug to return true?
> // Some ancestor nodes may not be in the latest snapshot.
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15319:
---
Attachment: HDFS-15319.000.patch
> Fix INode#isInLatestSnapshot() API
> --
>
> Key: HDFS-15319
> URL: https://issues.apache.org/jira/browse/HDFS-15319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15319.000.patch
>
>
> isInLatestSnapshot() may return true in cases where an inode's ancesstors
> might not be in the latest snapshot.
> {code:java}
> // if parent is a reference node, parent must be a renamed node. We can
> // stop the check at the reference node.
> if (parent != null && parent.isReference()) {
> // TODO: Is it a bug to return true?
> // Some ancestor nodes may not be in the latest snapshot.
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097210#comment-17097210
]
Shashikant Banerjee commented on HDFS-15319:
[~szetszwo], can you please have a look?
> Fix INode#isInLatestSnapshot() API
> --
>
> Key: HDFS-15319
> URL: https://issues.apache.org/jira/browse/HDFS-15319
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15319.000.patch
>
>
> isInLatestSnapshot() may return true in cases where an inode's ancesstors
> might not be in the latest snapshot.
> {code:java}
> // if parent is a reference node, parent must be a renamed node. We can
> // stop the check at the reference node.
> if (parent != null && parent.isReference()) {
> // TODO: Is it a bug to return true?
> // Some ancestor nodes may not be in the latest snapshot.
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15319:
---
Component/s: snapshots
> Fix INode#isInLatestSnapshot() API
> --
>
> Key: HDFS-15319
> URL: https://issues.apache.org/jira/browse/HDFS-15319
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15319.000.patch
>
>
> isInLatestSnapshot() may return true in cases where an inode's ancesstors
> might not be in the latest snapshot.
> {code:java}
> // if parent is a reference node, parent must be a renamed node. We can
> // stop the check at the reference node.
> if (parent != null && parent.isReference()) {
> // TODO: Is it a bug to return true?
> // Some ancestor nodes may not be in the latest snapshot.
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15313) Ensure inodes in active filesytem are not deleted during snapshot delete
[ https://issues.apache.org/jira/browse/HDFS-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097205#comment-17097205 ] Hudson commented on HDFS-15313: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18205 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18205/]) HDFS-15313. Ensure inodes in active filesytem are not deleted during (shashikant: rev 82343790eebc3ebe7ef81f6b89260e5bbf121d83) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/Diff.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java > Ensure inodes in active filesytem are not deleted during snapshot delete > > > Key: HDFS-15313 > URL: https://issues.apache.org/jira/browse/HDFS-15313 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15313.000.patch, HDFS-15313.001.patch > > > After HDFS-13101, it was observed in one of our customer deployments that > delete snapshot ends up cleaning up inodes from active fs which can be > referred from only one snapshot as the isLastReference() check for the parent > dir introduced in HDFS-13101 may return true in certain cases. The aim of > this Jira to add a check to ensure if the Inodes are being referred in the > active fs , should not get deleted while deletion of snapshot happens. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15313) Ensure inodes in active filesytem are not deleted during snapshot delete
[ https://issues.apache.org/jira/browse/HDFS-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097200#comment-17097200 ] Shashikant Banerjee commented on HDFS-15313: Filed HDFS-15319 to address the issues in isInLatestSnapshot(). > Ensure inodes in active filesytem are not deleted during snapshot delete > > > Key: HDFS-15313 > URL: https://issues.apache.org/jira/browse/HDFS-15313 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15313.000.patch, HDFS-15313.001.patch > > > After HDFS-13101, it was observed in one of our customer deployments that > delete snapshot ends up cleaning up inodes from active fs which can be > referred from only one snapshot as the isLastReference() check for the parent > dir introduced in HDFS-13101 may return true in certain cases. The aim of > this Jira to add a check to ensure if the Inodes are being referred in the > active fs , should not get deleted while deletion of snapshot happens. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDFS-15319:
--
Assignee: Shashikant Banerjee
> Fix INode#isInLatestSnapshot() API
> --
>
> Key: HDFS-15319
> URL: https://issues.apache.org/jira/browse/HDFS-15319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
>
> isInLatestSnapshot() may return true in cases where an inode's ancesstors
> might not be in the latest snapshot.
> {code:java}
> // if parent is a reference node, parent must be a renamed node. We can
> // stop the check at the reference node.
> if (parent != null && parent.isReference()) {
> // TODO: Is it a bug to return true?
> // Some ancestor nodes may not be in the latest snapshot.
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15315) IOException on close() when using Erasure Coding
[ https://issues.apache.org/jira/browse/HDFS-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097199#comment-17097199 ] Anshuman Singh commented on HDFS-15315: --- Hi [~weichiu] Referring to your previous comment. As HBase is built on top of HDFS, can I safely assume EC will work fine with HBase? Thanks > IOException on close() when using Erasure Coding > > > Key: HDFS-15315 > URL: https://issues.apache.org/jira/browse/HDFS-15315 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1, ec, hdfs >Affects Versions: 3.1.1 > Environment: XOR-2-1-1024k policy on hadoop 3.1.1 with 3 datanodes >Reporter: Anshuman Singh >Priority: Major > > When using Erasure Coding policy on a directory, the replication factor is > set to 1. Solr fails in indexing documents with error - _java.io.IOException: > Unable to close file because the last block does not have enough number of > replicas._ It works fine without EC (with replication factor as 3.) It seems > to be identical to this issue. [ > https://issues.apache.org/jira/browse/HDFS-11486|https://issues.apache.org/jira/browse/HDFS-11486] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15319:
---
Description:
isInLatestSnapshot() may return true in cases where an inode's ancesstors might
not be in the latest snapshot.
{code:java}
// if parent is a reference node, parent must be a renamed node. We can
// stop the check at the reference node.
if (parent != null && parent.isReference()) {
// TODO: Is it a bug to return true?
// Some ancestor nodes may not be in the latest snapshot.
return true;
}
{code}
was:
{code:java}
// if parent is a reference node, parent must be a renamed node. We can
// stop the check at the reference node.
if (parent != null && parent.isReference()) {
// TODO: Is it a bug to return true?
// Some ancestor nodes may not be in the latest snapshot.
return true;
}
{code}
> Fix INode#isInLatestSnapshot() API
> --
>
> Key: HDFS-15319
> URL: https://issues.apache.org/jira/browse/HDFS-15319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Priority: Major
>
> isInLatestSnapshot() may return true in cases where an inode's ancesstors
> might not be in the latest snapshot.
> {code:java}
> // if parent is a reference node, parent must be a renamed node. We can
> // stop the check at the reference node.
> if (parent != null && parent.isReference()) {
> // TODO: Is it a bug to return true?
> // Some ancestor nodes may not be in the latest snapshot.
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[
https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15319:
---
Description:
{code:java}
// if parent is a reference node, parent must be a renamed node. We can
// stop the check at the reference node.
if (parent != null && parent.isReference()) {
// TODO: Is it a bug to return true?
// Some ancestor nodes may not be in the latest snapshot.
return true;
}
{code}
was:The
> Fix INode#isInLatestSnapshot() API
> --
>
> Key: HDFS-15319
> URL: https://issues.apache.org/jira/browse/HDFS-15319
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Priority: Major
>
> {code:java}
> // if parent is a reference node, parent must be a renamed node. We can
> // stop the check at the reference node.
> if (parent != null && parent.isReference()) {
> // TODO: Is it a bug to return true?
> // Some ancestor nodes may not be in the latest snapshot.
> return true;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[ https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15319: --- Description: The > Fix INode#isInLatestSnapshot() API > -- > > Key: HDFS-15319 > URL: https://issues.apache.org/jira/browse/HDFS-15319 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Priority: Major > > The -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15319) Fix INode#isInLatestSnapshot
Shashikant Banerjee created HDFS-15319: -- Summary: Fix INode#isInLatestSnapshot Key: HDFS-15319 URL: https://issues.apache.org/jira/browse/HDFS-15319 Project: Hadoop HDFS Issue Type: Bug Reporter: Shashikant Banerjee -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15319) Fix INode#isInLatestSnapshot() API
[ https://issues.apache.org/jira/browse/HDFS-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15319: --- Summary: Fix INode#isInLatestSnapshot() API (was: Fix INode#isInLatestSnapshot) > Fix INode#isInLatestSnapshot() API > -- > > Key: HDFS-15319 > URL: https://issues.apache.org/jira/browse/HDFS-15319 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15313) Ensure inodes in active filesytem are not deleted during snapshot delete
[ https://issues.apache.org/jira/browse/HDFS-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15313: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~szetszwo] for the review. I have committed this. > Ensure inodes in active filesytem are not deleted during snapshot delete > > > Key: HDFS-15313 > URL: https://issues.apache.org/jira/browse/HDFS-15313 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15313.000.patch, HDFS-15313.001.patch > > > After HDFS-13101, it was observed in one of our customer deployments that > delete snapshot ends up cleaning up inodes from active fs which can be > referred from only one snapshot as the isLastReference() check for the parent > dir introduced in HDFS-13101 may return true in certain cases. The aim of > this Jira to add a check to ensure if the Inodes are being referred in the > active fs , should not get deleted while deletion of snapshot happens. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15302) Backport HDFS-15286 to branch-2.x
[ https://issues.apache.org/jira/browse/HDFS-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-15302: - Fix Version/s: 2.10.1 2.9.3 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) +1, committed to branch-2.10 and branch-2.9. Thanks [~hemanthboyina] for your contribution! > Backport HDFS-15286 to branch-2.x > - > > Key: HDFS-15302 > URL: https://issues.apache.org/jira/browse/HDFS-15302 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Akira Ajisaka >Assignee: hemanthboyina >Priority: Blocker > Fix For: 2.9.3, 2.10.1 > > Attachments: HDFS-15302-branch-2.10-01.patch, > HDFS-15302-branch-2.10-02.patch, HDFS-15302-branch.2.10.1.patch > > > Backport HDFS-15286 to branch-2.10 and branch-2.9. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
