[
https://issues.apache.org/jira/browse/HDFS-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010820#comment-14010820
]
Haohui Mai commented on HDFS-6382:
----------------------------------
bq. TTL is a very simple (but general) policy and we might even consider it as
an attribute of file, like the number of replicas. Seems it wouldn't introduce
much complexity to handle it in the NN.
bq. Another benefit to having it inside NN is we don't have to handle the
authentication/authorization problem in a separate system. For example we have
a shared HDFS cluster for many internal users, we don't want someone to set TTL
policy to other one's files. NN could handle it easily by its own
authentication/authorization mechanism.
I agree that running jobs of the namespace without MR should be the direction
to go. However, I think the main hold back here is that the design mixes the
mechanism (running jobs of the namespace without MR) and the policy (TTL)
together.
As [~cmccabe] pointed out earlier, every user has his / her own policy.
Provided that HDFS has a wide range of users, this type of design /
implementation is unlikely to fly in the ecosystem.
Currently HDFS does not have the above mechanism, you're more than welcomed to
contribute a patch.
> HDFS File/Directory TTL
> -----------------------
>
> Key: HDFS-6382
> URL: https://issues.apache.org/jira/browse/HDFS-6382
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
> Affects Versions: 2.4.0
> Reporter: Zesheng Wu
> Assignee: Zesheng Wu
>
> In production environment, we always have scenario like this, we want to
> backup files on hdfs for some time and then hope to delete these files
> automatically. For example, we keep only 1 day's logs on local disk due to
> limited disk space, but we need to keep about 1 month's logs in order to
> debug program bugs, so we keep all the logs on hdfs and delete logs which are
> older than 1 month. This is a typical scenario of HDFS TTL. So here we
> propose that hdfs can support TTL.
> Following are some details of this proposal:
> 1. HDFS can support TTL on a specified file or directory
> 2. If a TTL is set on a file, the file will be deleted automatically after
> the TTL is expired
> 3. If a TTL is set on a directory, the child files and directories will be
> deleted automatically after the TTL is expired
> 4. The child file/directory's TTL configuration should override its parent
> directory's
> 5. A global configuration is needed to configure that whether the deleted
> files/directories should go to the trash or not
> 6. A global configuration is needed to configure that whether a directory
> with TTL should be deleted when it is emptied by TTL mechanism or not.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
