[
https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057032#comment-14057032
]
Colin Patrick McCabe commented on HDFS-6634:
--------------------------------------------
I think it would be great to have some way for the client to tell the NameNode
"give me all the edits I have permission to see between transaction X and Y."
Then we could return them in a binary format (probably protobuf) that was
stable, compact, and stable over time (unlike the various textual logs we have
now).
This is something that people have asked for for a long time. Just offhand, I
can think of a lot of use cases:
* Hive/Impala/other SQL system: monitor for updates to tables via addition or
removal of HDFS files or directories
* Solr/ElasticSearch: start indexing HDFS files as soon as a client finishes
writing and closes them
* async replication / "log-shipping": we could start copying files to a backup
cluster as soon as they're closed.
It makes sense to do this in HDFS, just like its done inside ext4 and so forth.
We shouldn't need an external service to provide the basic API (although we
should make that possible, of course).
> inotify in HDFS
> ---------------
>
> Key: HDFS-6634
> URL: https://issues.apache.org/jira/browse/HDFS-6634
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs-client, namenode, qjm
> Reporter: James Thomas
> Assignee: James Thomas
> Attachments: inotify-intro.2.pdf, inotify-intro.pdf
>
>
> Design a mechanism for applications like search engines to access the HDFS
> edit stream.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
