[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120243#comment-16120243
]
Yongjun Zhang commented on HDFS-12217:
--
Hi [~manojg] and [~jojochuang],
Thanks for continuing to drive on this when I was out. I am glad to see that we
have converged on the solution.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch, HDFS-12217.04.patch, HDFS-12217.05.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111225#comment-16111225
]
Manoj Govindassamy commented on HDFS-12217:
---
Sorry for the inconvenience [~jlowe]. The cherrypick from the trunk and the
follow on build went through smoothly and I missed the test issue. Will rectify
the error. Thanks for the help.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch, HDFS-12217.04.patch, HDFS-12217.05.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110008#comment-16110008
]
Hudson commented on HDFS-12217:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12096 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12096/])
HDFS-12217. HDFS snapshots doesn't capture all open files when one of
(manojpec: rev 52d7bafcf49916887197436ddb0f08f021d248d9)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshotException.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestOpenFilesWithSnapshot.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch, HDFS-12217.04.patch, HDFS-12217.05.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109454#comment-16109454
]
Manoj Govindassamy commented on HDFS-12217:
---
Unit test failure and the findbug warnings are not related the patch.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch, HDFS-12217.04.patch, HDFS-12217.05.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109397#comment-16109397
]
Hadoop QA commented on HDFS-12217:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 56s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879864/HDFS-12217.05.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a05cc83d7ed2 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b38a1ee |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20517/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20517/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20517/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20517/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL:
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109206#comment-16109206
]
Wei-Chiu Chuang commented on HDFS-12217:
+1 pending Jenkins.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch, HDFS-12217.04.patch, HDFS-12217.05.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108886#comment-16108886
]
Wei-Chiu Chuang commented on HDFS-12217:
Thanks [~manojg] for the latest patch.
I am still a little concerned not logging the exception in
{{DirectorySnapshottableFeature#addSnapshot}}, because an exception could be
thrown from places other than {{LeaseManager#getINodeWithLeases}} (such as an
NPE in INodesInPath#getLastINode). If that happens, logging the exception
message wouldn't help much.
Can we add an overloaded SnapshotException constructor that has an additional
Throwable as an extra parameter, and throw that instead so that we pass that
exception to the client?
Other than this, the rest LGTM.
Thanks.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch, HDFS-12217.04.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108296#comment-16108296
]
Hadoop QA commented on HDFS-12217:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879752/HDFS-12217.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 425425fe3a9a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ea56812 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20507/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20507/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20507/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
>
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108147#comment-16108147
]
Wei-Chiu Chuang commented on HDFS-12217:
Thanks [~manojg] [~yzhangal].
One nit I found is that {{LeaseManager#getINodeWithLeases()}} seems to be a
test-only method. If so, it doesn't need public modifier (package private is
sufficient) and we can also add a {{@VisibleForTesting }} annotation.
On a separate note, and this is totally unrelated to this patch. It looks like
{{LeaseManager#addLease}} assumes the inodeId is an id for an INodeFile, which
makes perfect sense. The member variable {{leasesById}} also implicitly makes
the assumption that the id is for an INodeFile. However, there's no assertion
to ensure it is the case in the future. I wonder if it makes sense to add a new
{{addLease(String, INodeFile)}} method to make sure only INodeFile is passed
in. So it would go like:
{code}
Lease addLease(String holder, INodeFile inodeFile) {
addLease(holder, inodeFile.getId());
}
private synchronized Lease addLease(String holder, long inodeId) {
...
}
{code}
Note that in your patch, {{DirectorySnapshottableFeature#addSnapshot}} would
capture exception, but doesn't log the exception. I also think that in addition
to the snapshot name, you should print the snapshot root path for the error
message.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107758#comment-16107758
]
Manoj Govindassamy commented on HDFS-12217:
---
[~yzhangal]/[~jojochuang],
As per yongjun's suggestion, I modified {{SnapshotException}} to carry the
throwable along with the message. Yet the shell command doesn't display the
server exception stack trace as expected. As we discussed previously we can
take on this common shell-command error reporting model in a different jira.
Will wait for some more time for others to comment on patch v03. Thanks.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106182#comment-16106182
]
Manoj Govindassamy commented on HDFS-12217:
---
Thanks for the details [~yzhangal].
As you are aware, the NameNode is already logging all the exception and stack
traces with clear messages. *The one pending item we are discussing here is,
"createSnapshot" shell command to display the long NameNode error stack traces
in the console along with the error message.* I personally prefer the client
commands to show only meaningful detailed error messages and not the long error
stack traces. But heard from you that the general model in the hadoop shell
commands is to display the stack trace also. I tried the
{{SnapshotException(string, throwable)}} approach, and still the client shell
command doesn't see the stack trace. It only sees the error message. Can you
please point me to the shell command where we display the error stack traces
also to the console? I can try doing the same. Otherwise, as you recommended
will take up this into a new jira and can get the others committed. Thanks for
the review.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106143#comment-16106143
]
Yongjun Zhang commented on HDFS-12217:
--
Hi [~manojg],
{quote}
LeaseManager#getINodeWithLeases already logs the warning message with full
exception and stack trace.
{quote}
The logging you mentioned above happens at the server side and is about the
cause of the failure. What I hope to see is the exception chain at client side
when SnapshotException is thrown, so user can see immediately what caused the
exception in one place. Collecting the server side log to match to the client
symptom is a support effort which I hope we can avoid if possible.
Currently SnapshotException is:
{code}
public class SnapshotException extends IOException {
private static final long serialVersionUID = 1L;
public SnapshotException(final String message) {
super(message);
}
public SnapshotException(final Throwable cause) {
super(cause);
}
}
{code}
I examined all the places that call {{SnapshotException(final String
message)}}, none of them is called inside exception catch block. So the patch
here is the first time that we are throwing SnapshotException inside an
exception catch block. Interestingly, I don't see any places using
{{SnapshotException(final Throwable cause)}}. So it seems introducing a method
{{public SnapshotException(String message, Throwable cause)}} and use it in
this patch will not introduce inconsistency because it will be the first use of
it.
That's what I preferred. However, if you like to create a new jira to add the
new API, and change the single use which is introduced by the patch here, It's
ok with me too.
Thanks.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106076#comment-16106076
]
Manoj Govindassamy commented on HDFS-12217:
---
* {{LeaseManager#getINodeWithLeases}} already logs the warning message with
full exception and stack trace.
* The caller wraps the message into a {{SnapshotException}} for the upper layer
usage.
* SnapshotException() doesn't accept the throwable argument. We need to change
SnapshotException() and all the users in the all the snapshot commands to have
an uniform behavior of returning the throwable in the RPC call. Will file a new
jira for this.
[~yzhangal], are you ok with this ?
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106027#comment-16106027
]
Yongjun Zhang commented on HDFS-12217:
--
Thanks for the new rev [~manojg].
When a user see the failure, s/he might want to know more about what's the root
cause. Though some error is logged at server side, server side log may not be
immediately available. And it would be nice to see an cause-effect chain log
visible to user on the spot. This will reduce support effort. Agree that there
will be similar log at both server and client, but I think it doesn't really
hurt. So I still prefer having the {{e}} put into the cause of the exception.
Hope this makes sense to you.
Thanks.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106013#comment-16106013
]
Hadoop QA commented on HDFS-12217:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m
7s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879473/HDFS-12217.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d793814cb949 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 890e14c |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20484/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20484/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20484/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105976#comment-16105976
]
Hadoop QA commented on HDFS-12217:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879458/HDFS-12217.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 71ca42c72c8c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 890e14c |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20482/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20482/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20482/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20482/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105906#comment-16105906
]
Yongjun Zhang commented on HDFS-12217:
--
Thanks for the new rev [~manojg].
Some comments:
{quote}
But, f.get() can be null if the task didn't get to run properly.
{quote}
if the task didn't run properly, {{f.get()}} would throw an exception instead
of return null.
Suggest to put the {{e}} as the cause in the exception throwing.
{code}
205 } catch (Exception e) {
206 throw new SnapshotException("Failed to add snapshot: Unable to
" +
207 "capture all open files for the snapshot '" + name + "'.");
{code}
+1 after that pending jenkins.
Thanks.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105830#comment-16105830
]
Yongjun Zhang commented on HDFS-12217:
--
Hi [~manojg],
Thanks for reporting the issue and working on.
It looks good in general, some comments here:
1. we can use package wide scope for the isFileDeleted method instead of
protected
2. f can't be null in the code, I mean, either an exception will be thrown, or
f will valid value. So we don't need to check whether it's null here.
{code}
for (Future f : futureList) {
try {
if (f != null) {
iipSet.addAll(f.get());
}
} catch (Exception e) {
LOG.warn("INode filter task encountered exception: ", e);
}
}
...
{code}
3. If any exception is thrown in the above code, we can log a waning, and
rethrow the exception, because the exception likely indicate a real problem
that we don't expect to happen. If we continue, we might get inconsistent
snapshot.
Thanks.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12217.01.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105788#comment-16105788
]
Hadoop QA commented on HDFS-12217:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m
39s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879424/HDFS-12217.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 070bf080bb55 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 746189a |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20477/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20477/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20477/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj
