[
https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055611#comment-14055611
]
Andrew Wang commented on HDFS-6634:
-----------------------------------
Hi James,
I took a quick look through the doc, had a few high-level comments:
* Would be good to specify the usecases concretely. Would help us evaluate
which design is better.
* Might also be good to give some brief background about what inotify is, how
it's used in user apps historically, any differences between Linux and us in
terms of usecases
* Do we want the full-on inotify API (which means subscribing to certain events
on certain paths), or just exposing the entire edits stream to superusers? The
latter would be easier to do, might hit a bunch of our usecases. A full
inotify-like API could be done later as an enhancement.
* I think full-inotify is hard to do with just QJM, for the reasons you
mentioned. One additional thing I'd add is that QJM just sees bytes, so is
unable to interpret things like a path.
* If we wanted to offload this from the NN, it should be possible to have a
standalone mode sorta like the SbNN, which follows the edit stream but exposes
this new API.
* This might be an excuse to PB the edit log format, though there might be
performance concerns there.
Overall, I'm leaning toward putting this in the NN. It'd be cool to do the
non-voting quorum member for better latency, but I doubt that our latency
requirements are that stringent. We could also add a "long poll" API to QJM if
this does become an issue, which would be handy both for this and the SbNN
(since it'd mean fewer edits to replay on failover, and make stale reads less
stale).
> inotify in HDFS
> ---------------
>
> Key: HDFS-6634
> URL: https://issues.apache.org/jira/browse/HDFS-6634
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs-client, namenode, qjm
> Reporter: James Thomas
> Assignee: James Thomas
> Attachments: inotify-intro.pdf
>
>
> Design a mechanism for applications like search engines to access the HDFS
> edit stream.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
