[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Attachment: HDFS-10323.003.patch
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.6.0, 2.7.4, 3.0.0-beta1
>Reporter: Ben Podgursky
>Assignee: Wenxin He
> Attachments: HDFS-10323.001.patch, HDFS-10323.002.patch,
> HDFS-10323.003.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Status: Open (was: Patch Available)
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1, 2.7.4, 2.6.0
>Reporter: Ben Podgursky
>Assignee: Wenxin He
> Attachments: HDFS-10323.001.patch, HDFS-10323.002.patch,
> HDFS-10323.003.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Status: Patch Available (was: Open)
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1, 2.7.4, 2.6.0
>Reporter: Ben Podgursky
>Assignee: Wenxin He
> Attachments: HDFS-10323.001.patch, HDFS-10323.002.patch,
> HDFS-10323.003.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Status: Patch Available (was: Open)
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1, 2.7.4, 2.6.0
>Reporter: Ben Podgursky
>Assignee: Wenxin He
> Attachments: HDFS-10323.001.patch, HDFS-10323.002.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Attachment: HDFS-10323.002.patch
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.6.0, 2.7.4, 3.0.0-beta1
>Reporter: Ben Podgursky
>Assignee: Wenxin He
> Attachments: HDFS-10323.001.patch, HDFS-10323.002.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Attachment: (was: HDFS-10323.002.patch.txt)
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.6.0, 2.7.4, 3.0.0-beta1
>Reporter: Ben Podgursky
>Assignee: Wenxin He
> Attachments: HDFS-10323.001.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Attachment: HDFS-10323.002.patch.txt
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.6.0, 2.7.4, 3.0.0-beta1
>Reporter: Ben Podgursky
>Assignee: Wenxin He
> Attachments: HDFS-10323.001.patch, HDFS-10323.002.patch.txt
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Status: Open (was: Patch Available)
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1, 2.7.4, 2.6.0
>Reporter: Ben Podgursky
>Assignee: Wenxin He
> Attachments: HDFS-10323.001.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233962#comment-16233962
]
Wenxin He edited comment on HDFS-10323 at 11/1/17 11:51 AM:
I find this problem too when using spark. And undeleted files leading to HDFS
cluster no space left.
So according to [~bpodgursky]'s suggestion and [~cmccabe]'s comment
bq. 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then
all other FileSystems.
I submit 001 patch to fix the problem:
In this patch
# FileSystem.Cache.map changed to {color:red}LinkedHashmap{color} in which fs
are stored in {color:red}insertion order{color}.
When ViewFileSystem is initialized, DistributedFileSystem is first stored in
FileSystem.Cache.map and then ViewFileSystem.
# When FileSystem.Cache.closeAll invoke, all cached fs {color:red}close
inversely{color}, which like LiFO model. So ViewFileSystem close before its
referred DistributedFileSystems, and all deleteOnExit files will be deleted
safely before DistributedFileSystems close.
was (Author: vincent he):
I find this problem too when using spark. And undeleted files leading to HDFS
cluster no space left.
So according to [~bpodgursky]'s suggestion and [~cmccabe]'s comment
bq. 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then
all other FileSystems.
I submit 001 patch to fix the problem:
In this patch FileSystem.Cache.map changed to LinkedHashmap in which fs are
stored in insertion order.
When ViewFileSystem is initialized, DistributedFileSystem is first stored in
FileSystem.Cache.map and then ViewFileSystem.
When FileSystem.Cache.closeAll invoke, all cached fs close inversely, which
like LiFO model. So ViewFileSystem close before its referred
DistributedFileSystems, and all deleteOnExit files will be deleted safely
before DistributedFileSystems close.
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.6.0, 2.7.4, 3.0.0-beta1
>Reporter: Ben Podgursky
>Assignee: Wenxin He
>Priority: Major
> Attachments: HDFS-10323.001.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Attachment: HDFS-10323.001.patch
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.6.0, 2.7.4, 3.0.0-beta1
>Reporter: Ben Podgursky
>Assignee: Wenxin He
>Priority: Major
> Attachments: HDFS-10323.001.patch
>
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-10323:
-
Affects Version/s: 2.7.4
3.0.0-beta1
Status: Patch Available (was: Open)
I find this problem too when using spark. And undeleted files leading to HDFS
cluster no space left.
So according to [~bpodgursky]'s suggestion and [~cmccabe]'s comment
bq. 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then
all other FileSystems.
I submit 001 patch to fix the problem:
In this patch FileSystem.Cache.map changed to LinkedHashmap in which fs are
stored in insertion order.
When ViewFileSystem is initialized, DistributedFileSystem is first stored in
FileSystem.Cache.map and then ViewFileSystem.
When FileSystem.Cache.closeAll invoke, all cached fs close inversely, which
like LiFO model. So ViewFileSystem close before its referred
DistributedFileSystems, and all deleteOnExit files will be deleted safely
before DistributedFileSystems close.
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1, 2.7.4, 2.6.0
>Reporter: Ben Podgursky
>Assignee: Wenxin He
>Priority: Major
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10323) transient deleteOnExit failure in ViewFileSystem due to close() ordering
[
https://issues.apache.org/jira/browse/HDFS-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He reassigned HDFS-10323:
Assignee: Wenxin He
> transient deleteOnExit failure in ViewFileSystem due to close() ordering
>
>
> Key: HDFS-10323
> URL: https://issues.apache.org/jira/browse/HDFS-10323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.6.0
>Reporter: Ben Podgursky
>Assignee: Wenxin He
>Priority: Major
>
> After switching to using a ViewFileSystem, fs.deleteOnExit calls began
> failing frequently, displaying this error on failure:
> 16/04/21 13:56:24 INFO fs.FileSystem: Ignoring failure to deleteOnExit for
> path /tmp/delete_on_exit_test_123/a438afc0-a3ca-44f1-9eb5-010ca4a62d84
> Since FileSystem eats the error involved, it is difficult to be sure what the
> error is, but I believe what is happening is that the ViewFileSystem’s child
> FileSystems are being close()’d before the ViewFileSystem, due to the random
> order ClientFinalizer closes FileSystems; so then when the ViewFileSystem
> tries to close(), it tries to forward the delete() calls to the appropriate
> child, and fails because the child is already closed.
> I’m unsure how to write an actual Hadoop test to reproduce this, since it
> involves testing behavior on actual JVM shutdown. However, I can verify that
> while
> {code:java}
> fs.deleteOnExit(randomTemporaryDir);
> {code}
> regularly (~50% of the time) fails to delete the temporary directory, this
> code:
> {code:java}
> ViewFileSystem viewfs = (ViewFileSystem)fs1;
> for (FileSystem fileSystem : viewfs.getChildFileSystems()) {
> if (fileSystem.exists(randomTemporaryDir)) {
> fileSystem.deleteOnExit(randomTemporaryDir);
> }
>
}
> {code}
> always successfully deletes the temporary directory on JVM shutdown.
> I am not very familiar with FileSystem inheritance hierarchies, but at first
> glance I see two ways to fix this behavior:
> 1) ViewFileSystem could forward deleteOnExit calls to the appropriate child
> FileSystem, and not hold onto that path itself.
> 2) FileSystem.Cache.closeAll could first close all ViewFileSystems, then all
> other FileSystems.
> Would appreciate any thoughts of whether this seems accurate, and thoughts
> (or help) on the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12375) Fail to start/stop journalnodes using start-dfs.sh/stop-dfs.sh.
[
https://issues.apache.org/jira/browse/HDFS-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He updated HDFS-12375:
-
Attachment: hdfs-site.xml
bq. dfs.namenode.shared.edits.dir.mycluster or
dfs.namenode.shared.edits.dir=qjournal://y124.l42scl.hortonworks.com:8485/mycluster
?
The former.
In my situation, [^hdfs-site.xml]
{noformat}
dfs.namenode.shared.edits.dir.ns1
qjournal://hadoop3-0-0-01:8485;hadoop3-0-0-02:8485;hadoop3-0-0-03:8485;/ns1
{noformat}
{noformat}
root@hadoop3-0-0-01:/opt/hadoop# sbin/stop-dfs.sh
root@hadoop3-0-0-01:/opt/hadoop# jps
11648 Jps
2625 NodeManager
2299 ResourceManager
583 JournalNode
{noformat}
The JournalNode is still alive after sbin/stop-dfs.sh.
I have a investigate, maybe it's line 103 in sbin/stop-dfs.sh does not get the
value in this situation.
sbin/start-dfs.sh has the same problem.
{noformat}
100 #-
101 # quorumjournal nodes (if any)
102
103 SHARED_EDITS_DIR=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey
dfs.namenode.shared.edits.dir 2>&-)
104
105 case "${SHARED_EDITS_DIR}" in
106 qjournal://*)
{noformat}
> Fail to start/stop journalnodes using start-dfs.sh/stop-dfs.sh.
> ---
>
> Key: HDFS-12375
> URL: https://issues.apache.org/jira/browse/HDFS-12375
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation, scripts
>Affects Versions: 3.0.0-beta1
>Reporter: Wenxin He
>Assignee: Bharat Viswanadham
> Attachments: hdfs-site.xml
>
>
> When 'dfs.namenode.checkpoint.edits.dir' suffixed with the corresponding
> NameServiceID, we can not start/stop journalnodes using
> start-dfs.sh/stop-dfs.sh.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12375) Fail to start/stop journalnodes using start-dfs.sh/stop-dfs.sh.
[ https://issues.apache.org/jira/browse/HDFS-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenxin He reassigned HDFS-12375: Assignee: (was: Wenxin He) > Fail to start/stop journalnodes using start-dfs.sh/stop-dfs.sh. > --- > > Key: HDFS-12375 > URL: https://issues.apache.org/jira/browse/HDFS-12375 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation, scripts >Affects Versions: 3.0.0-beta1 >Reporter: Wenxin He > > When 'dfs.namenode.checkpoint.edits.dir' suffixed with the corresponding > NameServiceID, we can not start/stop journalnodes using > start-dfs.sh/stop-dfs.sh. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12374) Document the missing -ns option of haadmin.
[
https://issues.apache.org/jira/browse/HDFS-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenxin He reassigned HDFS-12374:
Assignee: (was: Wenxin He)
> Document the missing -ns option of haadmin.
> ---
>
> Key: HDFS-12374
> URL: https://issues.apache.org/jira/browse/HDFS-12374
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, federation
>Affects Versions: 3.0.0-alpha4
>Reporter: Wenxin He
>Priority: Minor
>
> Document the missing -ns option of haadmin in HDFSCommands.md,
> HDFSHighAvailabilityWithQJM.md and HDFSHighAvailabilityWithNFS.md.
> Before patch:
> {noformat}
> Usage:
> hdfs haadmin -transitionToActive [--forceactive]
> hdfs haadmin -transitionToStandby
> hdfs haadmin -failover [--forcefence] [--forceactive]
>
> hdfs haadmin -getServiceState
> hdfs haadmin -getAllServiceState
> hdfs haadmin -checkHealth
> hdfs haadmin -help
> {noformat}
> After patch:
> {noformat}
> Usage: haadmin [-ns ]
> [-transitionToActive [--forceactive] ]
> [-transitionToStandby ]
> [-failover [--forcefence] [--forceactive] ]
> [-getServiceState ]
> [-getAllServiceState]
> [-checkHealth ]
> [-help ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12375) Fail to start/stop journalnodes using start-dfs.sh/stop-dfs.sh.
Wenxin He created HDFS-12375: Summary: Fail to start/stop journalnodes using start-dfs.sh/stop-dfs.sh. Key: HDFS-12375 URL: https://issues.apache.org/jira/browse/HDFS-12375 Project: Hadoop HDFS Issue Type: Bug Components: federation, scripts Affects Versions: 3.0.0-beta1 Reporter: Wenxin He Assignee: Wenxin He When 'dfs.namenode.checkpoint.edits.dir' suffixed with the corresponding NameServiceID, we can not start/stop journalnodes using start-dfs.sh/stop-dfs.sh. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12374) Document the missing -ns option of haadmin.
Wenxin He created HDFS-12374:
Summary: Document the missing -ns option of haadmin.
Key: HDFS-12374
URL: https://issues.apache.org/jira/browse/HDFS-12374
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation, federation
Affects Versions: 3.0.0-alpha4
Reporter: Wenxin He
Assignee: Wenxin He
Priority: Minor
Document the missing -ns option of haadmin in HDFSCommands.md,
HDFSHighAvailabilityWithQJM.md and HDFSHighAvailabilityWithNFS.md.
Before patch:
{noformat}
Usage:
hdfs haadmin -transitionToActive [--forceactive]
hdfs haadmin -transitionToStandby
hdfs haadmin -failover [--forcefence] [--forceactive]
hdfs haadmin -getServiceState
hdfs haadmin -getAllServiceState
hdfs haadmin -checkHealth
hdfs haadmin -help
{noformat}
After patch:
{noformat}
Usage: haadmin [-ns ]
[-transitionToActive [--forceactive] ]
[-transitionToStandby ]
[-failover [--forcefence] [--forceactive] ]
[-getServiceState ]
[-getAllServiceState]
[-checkHealth ]
[-help ]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12125) Document the missing -removePolicy command of ec
[
https://issues.apache.org/jira/browse/HDFS-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086901#comment-16086901
]
Wenxin He commented on HDFS-12125:
--
Now usage of ec commands looks like this after regroup:
{noformat}
Usage: bin/hdfs ec [COMMAND]
[-listPolicies]
[-addPolicies -policyFile ]
[-enablePolicy -policy ]
[-disablePolicy -policy ]
[-removePolicy -policy ]
[-getPolicy -path ]
[-setPolicy -path -policy ]
[-unsetPolicy -path ]
[-listCodecs]
[-help ]
{noformat}
I am not sure whether this order is OK. Any comment is welcome. Thanks in
advance.
> Document the missing -removePolicy command of ec
>
>
> Key: HDFS-12125
> URL: https://issues.apache.org/jira/browse/HDFS-12125
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha4
>Reporter: Wenxin He
>Assignee: Wenxin He
> Attachments: HDFS-12125.001.patch
>
>
> Document the missing command -removePolicy in HDFSErasureCoding.md and
> HDFSCommands.md and regroup the ec commands to improve the user experience.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12125) Document the missing -removePolicy command of ec
[ https://issues.apache.org/jira/browse/HDFS-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085531#comment-16085531 ] Wenxin He commented on HDFS-12125: -- The test failure is irrelevant. > Document the missing -removePolicy command of ec > > > Key: HDFS-12125 > URL: https://issues.apache.org/jira/browse/HDFS-12125 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, erasure-coding >Affects Versions: 3.0.0-alpha4 >Reporter: Wenxin He >Assignee: Wenxin He > Attachments: HDFS-12125.001.patch > > > Document the missing command -removePolicy in HDFSErasureCoding.md and > HDFSCommands.md and regroup the ec commands to improve the user experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12125) Document the missing -removePolicy command of ec
[ https://issues.apache.org/jira/browse/HDFS-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenxin He updated HDFS-12125: - Status: Patch Available (was: Open) > Document the missing -removePolicy command of ec > > > Key: HDFS-12125 > URL: https://issues.apache.org/jira/browse/HDFS-12125 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, erasure-coding >Affects Versions: 3.0.0-alpha4 >Reporter: Wenxin He >Assignee: Wenxin He > Attachments: HDFS-12125.001.patch > > > Document the missing command -removePolicy in HDFSErasureCoding.md and > HDFSCommands.md and regroup the ec commands to improve the user experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12125) Document the missing -removePolicy command of ec
[ https://issues.apache.org/jira/browse/HDFS-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenxin He updated HDFS-12125: - Attachment: HDFS-12125.001.patch 001.patch: # rename 'policyName' to 'policy'. # document the missing -removePolicy command. # regroup the ec commands to improve the user experience. > Document the missing -removePolicy command of ec > > > Key: HDFS-12125 > URL: https://issues.apache.org/jira/browse/HDFS-12125 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, erasure-coding >Affects Versions: 3.0.0-alpha4 >Reporter: Wenxin He >Assignee: Wenxin He > Attachments: HDFS-12125.001.patch > > > Document the missing command -removePolicy in HDFSErasureCoding.md and > HDFSCommands.md and regroup the ec commands to improve the user experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12125) Document the missing -removePolicy command of ec
Wenxin He created HDFS-12125: Summary: Document the missing -removePolicy command of ec Key: HDFS-12125 URL: https://issues.apache.org/jira/browse/HDFS-12125 Project: Hadoop HDFS Issue Type: Bug Components: documentation, erasure-coding Affects Versions: 3.0.0-alpha4 Reporter: Wenxin He Assignee: Wenxin He Document the missing command -removePolicy in HDFSErasureCoding.md and HDFSCommands.md and regroup the ec commands to improve the user experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs' in doc
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057271#comment-16057271 ] wenxin he commented on HDFS-11978: -- I've closed it, thanks [~jojochuang]! > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' in doc > -- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Assignee: wenxin he >Priority: Minor > Fix For: 3.0.0-alpha4 > > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055218#comment-16055218 ] wenxin he commented on HDFS-11978: -- Thanks for adding me to the list, [~jojochuang]. BTW, anything shall I do to carry the work a step forward? Should I attach a patch corresponding to PR-237? or the PR link from github is enough for committing? I read this in "HowToContribute": bq. If there is a corresponding pull request, you don't need to attach a patch in this issue because the precommit job always runs on pull request instead of the attached patch. but I am not very sure about this. > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Assignee: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053837#comment-16053837 ] wenxin he commented on HDFS-11978: -- hi [~xiaochen], sorry to disturb you. Would you mind reviewing this patch? > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051633#comment-16051633 ] wenxin he commented on HDFS-11978: -- PR is ready, please kindly review. > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxin he updated HDFS-11978: - Attachment: (was: HDFS-11978.001.patch) > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxin he updated HDFS-11978: - Attachment: HDFS-11978.001.patch No Hadoop QA triggered for pull request, so attach a patch file > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > Attachments: HDFS-11978.001.patch > > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxin he updated HDFS-11978: - Status: Open (was: Patch Available) > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxin he updated HDFS-11978: - Status: Patch Available (was: Open) > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051402#comment-16051402 ] wenxin he commented on HDFS-11978: -- https://github.com/apache/hadoop/pull/237.patch > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051248#comment-16051248 ] wenxin he commented on HDFS-11978: -- https://github.com/apache/hadoop/pull/235.patch > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxin he updated HDFS-11978: - Status: Patch Available (was: Open) > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxin he updated HDFS-11978: - Status: Open (was: Patch Available) > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxin he updated HDFS-11978: - Status: Patch Available (was: Open) https://github.com/apache/hadoop/pull/235.patch > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
wenxin he created HDFS-11978: Summary: Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs' Key: HDFS-11978 URL: https://issues.apache.org/jira/browse/HDFS-11978 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 3.0.0-alpha4 Reporter: wenxin he Priority: Minor Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
