[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277839#comment-16277839
]
Junping Du commented on HDFS-12832:
---
Merge to branch-2.8.3 as well.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Fix For: 2.8.3, 2.7.5, 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269978#comment-16269978
]
Hudson commented on HDFS-12832:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13288 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13288/])
HDFS-12832. INode.getFullPathName may throw (shv: rev
d331762f24b3f22f609366740c9c4f449edc61ac)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ErasureCodingWork.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicationWork.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockReconstructionWork.java
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Fix For: 2.7.5, 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4
>
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269792#comment-16269792
]
Junping Du commented on HDFS-12832:
---
Thanks [~shv] for the patch and [~zhz] for review. I think we also need this
fix in branch-2.8 (branch-2.8.3) and branch-2.9. Adding 2.8.3, 2.9.1 and 3.0.0
in target version.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269744#comment-16269744
]
Zhe Zhang commented on HDFS-12832:
--
+1 on branch-2 and branch-2.7 patches as well. They are pretty clean fixes to
avoid race conditions on updating and reading {{bc}}'s members.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268446#comment-16268446
]
Zhe Zhang commented on HDFS-12832:
--
Thanks [~shv] for the fix. Trunk patch LGTM, +1. I'll finish reviewing branch
patches tomorrow.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266999#comment-16266999
]
Rushabh S Shah commented on HDFS-12832:
---
bq. I think if the name is calculated it must be inside the lock, otherwise
it's a bug.
Agreed. +1 non-binding.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266967#comment-16266967
]
Erik Krogen commented on HDFS-12832:
I think it is also relevant to point out that HDFS-10674 made that comment
somewhat less true as it makes the {{#getName()}} code path less expensive.
Certainly still worth considering but hopefully the impact is mitigated. I
agree with Konstantin that getting the safe fix in to unblock releases is the
right move, then a follow-on JIRA to measure performance implications of
removing {{srcPath}} from the API and depending on the results deprecating the
current API.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266312#comment-16266312
]
Konstantin Shvachko commented on HDFS-12832:
Hi [~shahrs87] this is what I called "dirty" fix, see [my earlier
comment|https://issues.apache.org/jira/browse/HDFS-12832?focusedCommentId=16263282=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16263282]
and communication afterwards. If you know the story behind the comment about
costly filename calculation please let us know. It's pretty old, I cannot see
beyond project merge point.
I think if the name is calculated it must be inside the lock, otherwise it's a
bug. We need to unblock upcoming releases, whichdepend on this issue, and the
safest way to fix this is by not changing any logic in block placement. Even
though in standard Hadoop placement policies file name is not involved we do
not know what other policies are out there. Also we cannot remove APIs before
they are deprecated.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266154#comment-16266154
]
Rushabh S Shah commented on HDFS-12832:
---
Thanks [~Deng FEI] for reporting the issue and [~shv] for the patch.
{noformat}
// choose replication targets: NOT HOLDING THE GLOBAL LOCK
// It is costly to extract the filename for which chooseTargets is called,
// so for now we pass in the Inode itself.
{noformat}
If we are not going to use the srcPath anywhere in
{{BlockPlacementPolicy#chooseTarget}}, then why to even compute
{{bc#getName()}} ?
The above comment was specifically made to move this computation outside the
lock and now we are moving this inside the lock and that too for the value that
is never going to get used.
IMO we should set the {{srcPath}} as null and add a warning in code to refer to
this jira so future programmers would know the background on why its set to
null.
In trunk, we can deprecate {{BlockPlacementPolicy.chooseTargets()}} with
{{srcPath}} parameter as you mentioned in HDFS-12856.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263903#comment-16263903
]
Erik Krogen commented on HDFS-12832:
Hey [~shv], changes look good to me and thank you for filing the two follow-on
JIRAs to address these rare but potentially disastrous race conditions. The
lack of additional unit test seems very reasonable to me. I wonder if there is
somewhere we should add developer documentation to indicate that this type of
usage pattern is not safe?
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263620#comment-16263620
]
Konstantin Shvachko commented on HDFS-12832:
TestUnbuffer is tracked under HDFS-12815. Other tests succeeded locally.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263596#comment-16263596
]
Hadoop QA commented on HDFS-12832:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} branch-2.7 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 188 unchanged - 1 fixed = 188 total (was 189) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
50s{color} | {color:red} The patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-hdfs:17 |
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeInitStorage |
| | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| Timed out junit tests | org.apache.hadoop.hdfs.server.datanode.TestHSync |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
|
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | org.apache.hadoop.hdfs.TestPread |
| | org.apache.hadoop.hdfs.TestDecommission |
| | org.apache.hadoop.hdfs.TestDFSAddressConfig |
| | org.apache.hadoop.hdfs.TestDFSUpgrade |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports |
| | org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache |
| | org.apache.hadoop.hdfs.TestDFSRollback |
| | org.apache.hadoop.hdfs.server.datanode.TestBlockScanner |
| | org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | org.apache.hadoop.hdfs.TestAbandonBlock |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:67e87c9 |
| JIRA Issue |
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263573#comment-16263573
]
Hadoop QA commented on HDFS-12832:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 24m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
34s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} branch-2.7 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 188 unchanged - 1 fixed = 188 total (was 189) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
56s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-hdfs:19 |
| Failed junit tests | hadoop.hdfs.TestClientReportBadBlock |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.web.TestWebHdfsTokens |
| Timed out junit tests | org.apache.hadoop.hdfs.TestSetrepDecreasing |
| | org.apache.hadoop.hdfs.TestFileAppend4 |
| | org.apache.hadoop.hdfs.TestRollingUpgradeDowngrade |
| | org.apache.hadoop.hdfs.TestReadWhileWriting |
| | org.apache.hadoop.hdfs.TestFileCorruption |
| | org.apache.hadoop.hdfs.TestLease |
| | org.apache.hadoop.hdfs.TestHDFSServerPorts |
| | org.apache.hadoop.hdfs.TestDFSUpgrade |
| | org.apache.hadoop.hdfs.web.TestWebHDFS |
| | org.apache.hadoop.hdfs.TestAppendSnapshotTruncate |
| | org.apache.hadoop.hdfs.TestRollingUpgradeRollback |
| | org.apache.hadoop.hdfs.TestFSOutputSummer |
| | org.apache.hadoop.hdfs.TestMiniDFSCluster |
| | org.apache.hadoop.hdfs.TestBlockReaderFactory |
| | org.apache.hadoop.hdfs.TestHFlush |
| | org.apache.hadoop.hdfs.TestEncryptedTransfer |
| | org.apache.hadoop.hdfs.TestDFSShell |
| | org.apache.hadoop.hdfs.TestDataTransferProtocol |
| | org.apache.hadoop.hdfs.TestHDFSTrash |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HDFS-12832 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263524#comment-16263524
]
Hadoop QA commented on HDFS-12832:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.fs.TestUnbuffer |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12832 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898935/HDFS-12832.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6f585d9ef76d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 738d1a2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22170/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22170/testReport/ |
| Max. process+thread count | 3882 (vs. ulimit of 5000) |
| modules | C:
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263499#comment-16263499
]
Konstantin Shvachko commented on HDFS-12832:
Also file HDFS-12856 to further address [~xkrogen]'s comment above, which still
remains valid after fixing {{ArrayIndexOutOfBoundException}} addressed in this
jira.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Assignee: Konstantin Shvachko
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-branch-2.002.patch,
> HDFS-12832-branch-2.7.002.patch, HDFS-12832-trunk-001.patch,
> HDFS-12832.002.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263296#comment-16263296
]
Konstantin Shvachko commented on HDFS-12832:
Filed HDFS-12855.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-trunk-001.patch, HDFS-12832.002.patch,
> exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262085#comment-16262085
]
Vinayakumar B commented on HDFS-12832:
--
bq. A more accurate approach is to remove ReplicationWork.bc and replace it
with two srcPath and storagePolicyID, which are the only things need from bc
and which could be computed inside ReplicationWork() from bc. The latter is
safe since it is constructed under the lock. That way we will avoid breaking
private interface BlockPlacementPolicy if it matters.
+1
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-trunk-001.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261880#comment-16261880
]
Konstantin Shvachko commented on HDFS-12832:
The funny thing is that in {{BlockPlacementPolicyDefault.chooseTarget(String
srcPath, ...)}} the {{srcPath}} parameter is completely redundant, as it is not
used at all.
I'm wondering if there are any policies outside that take file name into
account while choosing a target for a block?
So a dirty fix would be to pass {{src = null}} and avoid calling
{{bc.getName()}} completely:
{code:java}
private void chooseTargets(BlockPlacementPolicy blockplacement,
BlockStoragePolicySuite storagePolicySuite,
Set excludedNodes) {
try {
- targets = blockplacement.chooseTarget(bc.getName(),
+ targets = blockplacement.chooseTarget(null,
additionalReplRequired, srcNode, liveReplicaStorages, false,
excludedNodes, block.getNumBytes(),
storagePolicySuite.getPolicy(bc.getStoragePolicyID()));
{code}
A more accurate approach is to remove {{ReplicationWork.bc}} and replace it
with two {{srcPath}} and {{storagePolicyID}}, which are the only things need
from {{bc}} and which could be computed inside {{ReplicationWork()}} from
{{bc}}. The latter is safe since it is constructed under the lock. That way we
will avoid breaking *private* interface {{BlockPlacementPolicy}} if it matters.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-12832-trunk-001.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260198#comment-16260198
]
DENG FEI commented on HDFS-12832:
-
[~xkrogen]
Has been uploaded on the stack, it happened at
{{ReplicationWork#chooseTargets()}} indeed.
And you are right. {{INode#getPathComponents()}} has same problem when
concurrently do {{move}} but not only {{rename}}.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Priority: Critical
> Attachments: HDFS-12832-trunk-001.patch, exception.log
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259877#comment-16259877
]
Konstantin Shvachko commented on HDFS-12832:
[~Deng FEI] could you please post the actual exception here if you have it. It
would be good to see a stack trace.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Priority: Critical
> Attachments: HDFS-12832-trunk-001.patch
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12832) INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit
[
https://issues.apache.org/jira/browse/HDFS-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259458#comment-16259458
]
Erik Krogen commented on HDFS-12832:
Thanks for reporting this and for working on a patch, [~Deng FEI]]! Actually,
the new method you are using, {{INode#getPathComponents()}} is subject to the
same race condition. Generally {{INode}} is not meant to be a concurrent data
structure as far as I can tell. I believe the issue is actually that
{{ReplicationWork#chooseTargets()}} is being called without a lock:
{code:title=BlockManager.ReplicationWork}
// choose replication targets: NOT HOLDING THE GLOBAL LOCK
// It is costly to extract the filename for which chooseTargets is called,
// so for now we pass in the block collection itself.
rw.chooseTargets(blockplacement, storagePolicySuite, excludedNodes);
{code}
Within {{chooseTargets()}} various methods on {{INode}}/{{BlockCollection}},
{{DatanodeDescriptor}}, {{DatanodeStorageInfo}}, and {{Block}} are called which
it seems should not be allowed outside of the lock.
[~Deng FEI], do you have a stack trace available to confirm that this is the
same code path which caused your exception? This is the code path that was
taken to trigger the issue for us.
> INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to
> NameNode exit
>
>
> Key: HDFS-12832
> URL: https://issues.apache.org/jira/browse/HDFS-12832
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.4, 3.0.0-beta1
>Reporter: DENG FEI
>Priority: Critical
> Attachments: HDFS-12832-trunk-001.patch
>
>
> {code:title=INode.java|borderStyle=solid}
> public String getFullPathName() {
> // Get the full path name of this inode.
> if (isRoot()) {
> return Path.SEPARATOR;
> }
> // compute size of needed bytes for the path
> int idx = 0;
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> // add component + delimiter (if not tail component)
> idx += inode.getLocalNameBytes().length + (inode != this ? 1 : 0);
> }
> byte[] path = new byte[idx];
> for (INode inode = this; inode != null; inode = inode.getParent()) {
> if (inode != this) {
> path[--idx] = Path.SEPARATOR_CHAR;
> }
> byte[] name = inode.getLocalNameBytes();
> idx -= name.length;
> System.arraycopy(name, 0, path, idx, name.length);
> }
> return DFSUtil.bytes2String(path);
> }
> {code}
> We found ArrayIndexOutOfBoundsException at
> _{color:#707070}System.arraycopy(name, 0, path, idx, name.length){color}_
> when ReplicaMonitor work ,and the NameNode will quit.
> It seems the two loop is not synchronized, the path's length is changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
