[
https://issues.apache.org/jira/browse/METRON-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otto Fowler updated METRON-534:
-------------------------------
Priority: Major (was: Minor)
> Metron Should have service to monitor HDFS for changes to files
> ---------------------------------------------------------------
>
> Key: METRON-534
> URL: https://issues.apache.org/jira/browse/METRON-534
> Project: Metron
> Issue Type: New Feature
> Reporter: Otto Fowler
> Assignee: Otto Fowler
> Priority: Major
> Attachments: inotify-ascii.png
>
>
> With the INotify library it is possible to receive notifications from HDFS on
> file changes.
> A service that utilized this and was configured with "rules" to process
> notifications on files of interest ( configurations, rules etc ) in the
> system would be able to send notifications to Zookeeper
> This may allow for things that are better stored in HDFS to remain there,
> and not force the system to put things in Zookeeper just for change
> notifications.
> Issues:
> * Inotify requires hdfs super user, what are the implications on a secure
> cluster etc?
> * performance / scalability and load
> * Can it be deployed with yarn?
> * Is Zookeeper actually the right place for all the configurations
> ---------------------------------------------------------------------
>
> UPDATE: 1/29/18
> "
> In the end, what I’m thinking is this:
>
> We have an ambari service that runs the notification -> zookeeper
> it reads the ‘registration area’ from zookeeper to get it’s state and what to
> watch
> post 777 when parsers are installed and registered it is trivial to have my
> installer also register the files to watch
>
> the notifications service also has a notification from zookeeper for new
> registrations.
>
> On notify event, the ‘notification node’ has it’s content set to the event
> details and time
> which the parser would pick up…. causing the reload
> "
>
> POC:
> [hdfs-inotify-zookeeper|https://github.com/ottobackwards/hdfs-inotify-zookeeper]
>
> !inotify-ascii.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
