[
https://issues.apache.org/jira/browse/HDFS-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011305#comment-14011305
]
Chris Nauroth commented on HDFS-6382:
-------------------------------------
bq. ...run a job (maybe periodically) over the namespace inside the NN...
Please correct me if I misunderstood, but this sounds like execution of
arbitrary code inside the NN process. If so, this opens the risk of resource
exhaustion at the NN by buggy or malicious code. Even if there is a fork for
process isolation, it's still sharing machine resources with the NN process.
If the code is running as the HDFS super-user, then it has access to sensitive
resources like the fsimage file. If multiple such "in-process jobs" are
submitted concurrently, then it would cause resource contention with the main
work of the NN. Multiple concurrent jobs also gets into the realm of
scheduling. There are lots of tough problems here that would increase the
complexity of the NN.
Even putting that aside, I see multiple advantages in implementing this
externally instead of embedded inside the NN. Here is a list of several
problems that an embedded design would need to solve, and which I believe are
already easily addressed by an external design. This includes/expands on
issues brought up by others in earlier comments too.
* Trash: The description mentions trash capability as a requirement. Trash
functionality is currently implemented as a client-side capability.
** Embedded: We'd need to reimplement trash inside the NN, or heavily refactor
for code sharing.
** External: The client already has the trash capability, so this problem is
already solved.
* Integration: Many Hadoop deployments use an alternative file system like S3
or Azure storage. In these deployments, there is no NameNode.
** Embedded: The feature is only usable for HDFS-based deployments. Users of
alternative file systems can't use the feature.
** External: The client already has the capability to target any Hadoop file
system implementation, so this problem is already solved.
* HA: In the event of a failover, we must guarantee that the former active NN
does not drive any expiration activity.
** Embedded: Any background thread or "in-process jobs" running inside the NN
must coordinate shutdown during a failover.
** External: Thanks to our client-side retry policies, an external process
automatically transitions to the new active NN after a failover, and there is
no risk of split-brain scenario, so this problem is already solved.
* Authentication/Authorization: Who exactly is the effective user running the
delete, and how do we manage their login and file permission enforcement?
** Embedded: You mention there is an advantage to running embedded, but I
didn't quite understand. Are you suggesting running the deletes inside a
{{UserGroupInformation#doAs}} for the specific user?
** External: The client already knows how to authenticate RPC, and the NN
already knows how to enforce authorization on files for that authenticated
user, so this problem is already solved.
* Error Handling: How do users find out when the deletes don't work?
** Embedded: There is no mechanism for asynchronous user notification inside
the NN. As others have mentioned, there is a lot of complexity in this area.
If it's email, then you need to solve the problem of reliable email delivery
(i.e. retries if SMTP gateways are down). If it's monitoring/alerting, then
you need to expose new monitoring endpoints to publish sufficient information.
** External: The client's exception messages are sufficient to identify file
paths that failed during synchronous calls, and the NN audit log is another
source of troubleshooting information, so this problem is already solved.
* Federation: With federation, the HDFS namespace is split across multiple
NameNodes.
** Embedded: The design needs to coordinate putting the right expiration work
on the right NN hosting that part of the namespace.
** External: The client has the capability to configure a client-side mount
table that joins together multiple federated namespaces, and {{ViewFileSystem}}
then routes RPC to the correct NN depending on the target file path, so this
problem is already solved.
> HDFS File/Directory TTL
> -----------------------
>
> Key: HDFS-6382
> URL: https://issues.apache.org/jira/browse/HDFS-6382
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
> Affects Versions: 2.4.0
> Reporter: Zesheng Wu
> Assignee: Zesheng Wu
>
> In production environment, we always have scenario like this, we want to
> backup files on hdfs for some time and then hope to delete these files
> automatically. For example, we keep only 1 day's logs on local disk due to
> limited disk space, but we need to keep about 1 month's logs in order to
> debug program bugs, so we keep all the logs on hdfs and delete logs which are
> older than 1 month. This is a typical scenario of HDFS TTL. So here we
> propose that hdfs can support TTL.
> Following are some details of this proposal:
> 1. HDFS can support TTL on a specified file or directory
> 2. If a TTL is set on a file, the file will be deleted automatically after
> the TTL is expired
> 3. If a TTL is set on a directory, the child files and directories will be
> deleted automatically after the TTL is expired
> 4. The child file/directory's TTL configuration should override its parent
> directory's
> 5. A global configuration is needed to configure that whether the deleted
> files/directories should go to the trash or not
> 6. A global configuration is needed to configure that whether a directory
> with TTL should be deleted when it is emptied by TTL mechanism or not.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
