[
https://issues.apache.org/jira/browse/HDFS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345892#comment-15345892
]
Ming Ma commented on HDFS-8940:
-------------------------------
[~zhz], thanks for the follow-up. Shortly after posting the design, we
de-prioritized the work and haven't done anything. It is great you can pick it
up.
Regarding using Kafka, your suggestion of building an independent app is a good
idea. At least you can try it out. Our case might be somewhat different from
yours, where we have large number of clients pulling infrequently changed
files; thus the question if Kafka can scale to handle broadcast to large number
of consumers.
After you get it working, it might be useful to abstract the underlying
notification channel, so that others can use other systems besides Kafka if
necessary.
To quantify the benefit, maybe we can measure the write performance, # of
blocks created per sec.
When applications use the service, is it going to be similar to the existing
DFSInotifyEventInputStream?
> Support for large-scale multi-tenant inotify service
> ----------------------------------------------------
>
> Key: HDFS-8940
> URL: https://issues.apache.org/jira/browse/HDFS-8940
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ming Ma
> Attachments: Large-Scale-Multi-Tenant-Inotify-Service.pdf
>
>
> HDFS-6634 provides the core inotify functionality. We would like to extend
> that to provide a large-scale service that ten of thousands of clients can
> subscribe to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
