[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061643#comment-16061643
]
Sean Mackrory commented on HADOOP-14499:
Filed HADOOP-14585 for that, as it's unrelated to the findbugs warning other
ways that line was incorrect. Any objection to committing the patch with the
javadocs change removed?
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch, HADOOP-14499-HADOOP-13345.003.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060937#comment-16060937
]
Steve Loughran commented on HADOOP-14499:
-
temp files work on a shared FS. AWS is actually somewhat sensitive to clocks:
if your VM is too far out of time then auth actually fails, its ~+-15 minutes.
There's some stuff in the Java SDK to actually calculate and adjust clock skew,
presumably parsing the timestamp of a failure, calculating the difference and
retrying. Which means that the field in {{SDKGlobalConfiguration}} could help
identify the difference between local time and AWS time.
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch, HADOOP-14499-HADOOP-13345.003.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059709#comment-16059709
]
Sean Mackrory commented on HADOOP-14499:
Just noticed there are actually 2 findbugs entries, and the second one does
actually show the issue as fixed.
{quote}bear in mind that we can't guarantee that the clocks of all clients are
in sync; you don't want a client whose TZ setting is wrong to aggressively
prune things. Had that happen in production with files in shared filestore.
This is why ant -diagnostics checks time consistency with temp files...
{quote}
Good point - we should at least have a big warning. How would you use temp
files to get a reference time to compare output of currentTimeMillis against?
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch, HADOOP-14499-HADOOP-13345.003.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059633#comment-16059633
]
Hadoop QA commented on HADOOP-14499:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
48s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red} hadoop-tools/hadoop-aws in HADOOP-13345 has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} hadoop-tools/hadoop-aws generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14499 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874103/HADOOP-14499-HADOOP-13345.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux dfb41beace8c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / 0db7176 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12605/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-aws-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12605/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12605/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch, HADOOP-14499-HADOOP-13345.003.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059124#comment-16059124
]
Steve Loughran commented on HADOOP-14499:
-
bear in mind that we can't guarantee that the clocks of all clients are in
sync; you don't want a client whose TZ setting is wrong to aggressively prune
things. Had that happen in production with files in shared filestore. This is
why {{ant -diagnostics}} checks time consistency with temp files...
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058469#comment-16058469
]
Aaron Fabbri commented on HADOOP-14499:
---
Yeah sorry for delay responding to your earlier comment. I think we need a
more rigorous definition of prune(), at least for my understanding. Would be
nice to write up the rules for isAuthoritative (you have to clear bit on parent
if you prune a child, etc) and have more contract tests around it.
Personally I'd just fix the findbugs warning in this JIRA. :-)
My previous concern was in deleting empty directories regardless of age: They
do need to be present during the inconsistency window. i.e.
getFileStatus(empty-dir-path) should see empty directories in MetadataStore
until the consistency window (prune age) has elapsed. Maybe I misread your
original patch?
{quote}
Since we currently don't track the last time at which a directory listing was
modified
{quote}
Couldn't a MetadataStore track the time the entry was last written into the
MetadataStore?
There are two "modification" times we can consider:
A. The time which an entry was (last) written into a MetadataStore
B. The FileStatus's modification time field.
My thinking is that prune is based on A (mostly because B is unreliable): The
sliding inconsistency window. It works nicely for both inconsistency and
caching (unlike B). For files, we can use the fact that A>=B.
Anyways, it seems to me that a MetadataStore could safely expire and/or prune
directory metadata, so I'm not sure the comment change is needed?
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058397#comment-16058397
]
Sean Mackrory commented on HADOOP-14499:
That's from my comment on Jun 8th. Since we currently don't track the last time
at which a directory listing was modified and since it inherently impacts
entries other than itself, there's no way for an implementation to safely prune
directories without raising the concerns you had about doing it here, right?
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058374#comment-16058374
]
Aaron Fabbri commented on HADOOP-14499:
---
Hey [~mackrorysd], thank you for the patch. Can you elaborate on this part:
{noformat}
- * Implementations MUST clear file metadata, and MAY clear directory metadata
- * (s3a itself does not track modification time for directories).
+ * Implementations MUST clear file metadata, but MUST NOT clear directory
+ * metadata (s3a itself does not track modification time for directories).
{noformat}
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058021#comment-16058021
]
Steve Loughran commented on HADOOP-14499:
-
My IDE says that {{ancestors}} is the only unused field too.
+1 for the patch, let's see what yetus says *after*
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057721#comment-16057721
]
Sean Mackrory commented on HADOOP-14499:
I'm really quite mystified by that findbugs continuing to show up. The patch
passed Yetus locally and clearly removes that variable at the line mentioned...
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056591#comment-16056591
]
Hadoop QA commented on HADOOP-14499:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
54s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} hadoop-tools/hadoop-aws in HADOOP-13345 has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} hadoop-tools/hadoop-aws generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14499 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873329/HADOOP-14499-HADOOP-13345.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5456467e15e3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / 0db7176 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12579/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-aws-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12579/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12579/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051690#comment-16051690
]
Steve Loughran commented on HADOOP-14499:
-
We should have the local version match the remote version as much as possible,
just for consistency in testing
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050600#comment-16050600
]
Steve Loughran commented on HADOOP-14499:
-
FWIW, the findbugs change is from HADOOP-14316
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043602#comment-16043602
]
Sean Mackrory commented on HADOOP-14499:
Well the idea behind prune is to not store metadata indefinitely if one is more
interested in the short-term consistency than the longer-term performance
improvements at the cost of storing all metadata for stuff that hasn't been
modified in days, weeks, etc. In some ways this is less of a concern with
LocalMetadatastore because the data only persists as long as the JVM it's in,
and it's not intended for production. But in some ways it's a little worse
because then it's not just a cost issue it's also an issue of how much memory
you're using for all that metadata. All in all, probably not a huge deal.
{quote}yeah, shouldn't we be keeping dirs until they are actually
deleted?{quote}
Side note: even then they don't get deleted, but the tombstones are just the
same as files for the sake of this discussion - they're only used to mark that
a certain path was deleted. So they should still be cleaned up in a prune. The
time of deletion is listed as the tombstones mod_time, so if you do a prune of
everything older than 24 hours, stuff that was deleted 24+ hours ago should now
*actually* be deleted from memory. Which is exactly what we want: tombstones
have no purpose once S3 reconciles all it's listing with the deletion.
We could (and probably should) just do what the DynamoDB implementation
currently does and leave all directories in place. But we should still at least
update parent directory listings to not include references to those files and
mark them as not-authoritative. The other option is that we don't prune
directories that were empty to begin with (may be brand new), but we do prune
directories that haven't had their contents changed at all in the time period
of the prune (i.e. if a directories entire contents was just pruned, and has no
other contents, we prune it). It's still an assumption though that something
else isn't in the process of adding a new child to that directory.
The last time we discussed more formal semantics for prune when it was being
developed was that files MUST be pruned, directories MAY be pruned. Since it
doesn't seem that we can ever make a completely safe assumption that a
directory's metadata is well and truly stale, do we want to change that to
files MUST be pruned and directories MUST NOT be pruned, ever? And then just do
the pruning in Local we were doing before but add something to reset
isAuthoritative on the parent directory whenever a file gets pruned?
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043139#comment-16043139
]
Steve Loughran commented on HADOOP-14499:
-
yeah, shouldn't we be keeping dirs until they are actually deleted?
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042079#comment-16042079
]
Aaron Fabbri commented on HADOOP-14499:
---
I feel like we need to define prune() semantics a little better, along with the
algorithm that satisfies them.. I'll admit I did not carefully review the
LocalMetadataStore's prune() implementation before, since it is not really used
in production (just for test).
After applying this patch, for example, I'm not sure why we remove directories
when they are empty. An empty directory is a valid thing to track but this
function seems to discard them regardless of age. Maybe you can describe the
intended algorithm a bit better, or we can discuss in person and post our notes
to this JIRA?
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
