[jira] [Commented] (HDFS-6526) Implement HDFS TtlManager
[
https://issues.apache.org/jira/browse/HDFS-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806189#comment-15806189
]
Hadoop QA commented on HDFS-6526:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-6526 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-6526 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12651960/HDFS-6526.1.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18092/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Implement HDFS TtlManager
> -
>
> Key: HDFS-6526
> URL: https://issues.apache.org/jira/browse/HDFS-6526
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6526.1.patch
>
>
> This issue is used to track development of HDFS TtlManager, for details see
> HDFS-6382.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6526) Implement HDFS TtlManager
[
https://issues.apache.org/jira/browse/HDFS-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524917#comment-14524917
]
Hadoop QA commented on HDFS-6526:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12651960/HDFS-6526.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10656/console |
This message was automatically generated.
> Implement HDFS TtlManager
> -
>
> Key: HDFS-6526
> URL: https://issues.apache.org/jira/browse/HDFS-6526
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6526.1.patch
>
>
> This issue is used to track development of HDFS TtlManager, for details see
> HDFS-6382.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6526) Implement HDFS TtlManager
[
https://issues.apache.org/jira/browse/HDFS-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524868#comment-14524868
]
Hadoop QA commented on HDFS-6526:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12651960/HDFS-6526.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10637/console |
This message was automatically generated.
> Implement HDFS TtlManager
> -
>
> Key: HDFS-6526
> URL: https://issues.apache.org/jira/browse/HDFS-6526
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6526.1.patch
>
>
> This issue is used to track development of HDFS TtlManager, for details see
> HDFS-6382.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6526) Implement HDFS TtlManager
[ https://issues.apache.org/jira/browse/HDFS-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075238#comment-14075238 ] Zesheng Wu commented on HDFS-6526: -- Thanks for feedback, [~daryn]. # About scalability and performance issues: currently the default implementation just simply traverses the whole directory tree, we can improve this to that each instance traverses a sub directory tree, and can deploy multiple instances on demand. # About bugs: can you point out in details, I would like to figure them out as soon as possible. # bq. Would you please elaborate on whether you plan to simply have the trash emptier and ttl manager run as distinct services in the same adjunct process? Or do you plan on the emptier actually leveraging/relying on ttls? As I mentioned in the design document in HDFS-6382, we want to supply a general mechanism that can run various kinds of policies on the namespace, for example TTL is one of the policies which is intended to clean expired files and directories. In the same way, we can extend to implement a trash policy to work in the way that is the same as the current trash emptier. If folks come to an agreement that move the trash emptier is reasonable, we will change 'TtlManager' to a more general name. # bq. As a general comment, a feature like this is attractive but may be highly dangerous. You many want to consider means to safeguard against a severely skewed system clock, else the ttl manager might go on mass murder spree in the filesystem Yes, I agree with you on this very much. The current patch is just an initial implementation, we will improve to support this in the future iteration. Hope that I expressed myself clearly:) > Implement HDFS TtlManager > - > > Key: HDFS-6526 > URL: https://issues.apache.org/jira/browse/HDFS-6526 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode >Affects Versions: 2.4.0 >Reporter: Zesheng Wu >Assignee: Zesheng Wu > Attachments: HDFS-6526.1.patch > > > This issue is used to track development of HDFS TtlManager, for details see > HDFS-6382. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6526) Implement HDFS TtlManager
[ https://issues.apache.org/jira/browse/HDFS-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073551#comment-14073551 ] Daryn Sharp commented on HDFS-6526: --- I see scalability and performance issues (maybe a few bugs) with the ttl manager which make it unsuitable for large namesystems. I wasn't too concerned because it's a "nice to have feature". I perked up when folding in the "must have" trash emptier was mentioned in HDFS-6525. Would you please elaborate on whether you plan to simply have the trash emptier and ttl manager run as distinct services in the same adjunct process? Or do you plan on the emptier actually leveraging/relying on ttls? As a general comment, a feature like this is attractive but may be highly dangerous. You many want to consider means to safeguard against a severely skewed system clock, else the ttl manager might go on mass murder spree in the filesystem... > Implement HDFS TtlManager > - > > Key: HDFS-6526 > URL: https://issues.apache.org/jira/browse/HDFS-6526 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode >Affects Versions: 2.4.0 >Reporter: Zesheng Wu >Assignee: Zesheng Wu > Attachments: HDFS-6526.1.patch > > > This issue is used to track development of HDFS TtlManager, for details see > HDFS-6382. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6526) Implement HDFS TtlManager
[
https://issues.apache.org/jira/browse/HDFS-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040792#comment-14040792
]
Hadoop QA commented on HDFS-6526:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651960/HDFS-6526.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1264 javac
compiler warnings (more than the trunk's current 1259 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.ttlmanager.TestTtlPolicy
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7212//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7212//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7212//console
This message is automatically generated.
> Implement HDFS TtlManager
> -
>
> Key: HDFS-6526
> URL: https://issues.apache.org/jira/browse/HDFS-6526
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6526.1.patch
>
>
> This issue is used to track development of HDFS TtlManager, for details see
> HDFS-6382.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
