[
https://issues.apache.org/jira/browse/METRON-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otto Fowler updated METRON-534:
-------------------------------
Description:
With the INotify library it is possible to receive notifications from HDFS on
file changes.
A service that utilized this and was configured with "rules" to process
notifications on files of interest ( configurations, rules etc ) in the system
would be able to send notifications to Zookeeper
This may allow for things that are better stored in HDFS to remain there, and
not force the system to put things in Zookeeper just for change notifications.
Issues:
* Inotify requires hdfs super user, what are the implications on a secure
cluster etc?
* performance / scalability and load
* Can it be deployed with yarn?
* Is Zookeeper actually the right place for all the configurations
---------------------------------------------------------------------
UPDATE: 1/29/18
"
In the end, what I’m thinking is this:
We have an ambari service that runs the notification -> zookeeper
it reads the ‘registration area’ from zookeeper to get it’s state and what to
watch
post 777 when parsers are installed and registered it is trivial to have my
installer also register the files to watch
the notifications service also has a notification from zookeeper for new
registrations.
On notify event, the ‘notification node’ has it’s content set to the event
details and time
which the parser would pick up…. causing the reload
"
POC:
[hdfs-inotify-zookeeper|https://github.com/ottobackwards/hdfs-inotify-zookeeper]
!inotify-ascii.png!
was:
With the INotify library it is possible to receive notifications from HDFS on
file changes.
A service that utilized this and was configured with "rules" to process
notifications on files of interest ( configurations, rules etc ) in the system
would be able to send notifications to Zookeeper
This may allow for things that are better stored in HDFS to remain there, and
not force the system to put things in Zookeeper just for change notifications.
Issues:
* Inotify requires hdfs super user, what are the implications on a secure
cluster etc?
* performance / scalability and load
* Can it be deployed with yarn?
* Is Zookeeper actually the right place for all the configurations
---------------------------------------------------------------------
!inotify-ascii.png!
> Metron Should have service to monitor HDFS for changes to files
> ---------------------------------------------------------------
>
> Key: METRON-534
> URL: https://issues.apache.org/jira/browse/METRON-534
> Project: Metron
> Issue Type: New Feature
> Reporter: Otto Fowler
> Assignee: Otto Fowler
> Priority: Minor
> Attachments: inotify-ascii.png
>
>
> With the INotify library it is possible to receive notifications from HDFS on
> file changes.
> A service that utilized this and was configured with "rules" to process
> notifications on files of interest ( configurations, rules etc ) in the
> system would be able to send notifications to Zookeeper
> This may allow for things that are better stored in HDFS to remain there,
> and not force the system to put things in Zookeeper just for change
> notifications.
> Issues:
> * Inotify requires hdfs super user, what are the implications on a secure
> cluster etc?
> * performance / scalability and load
> * Can it be deployed with yarn?
> * Is Zookeeper actually the right place for all the configurations
> ---------------------------------------------------------------------
>
> UPDATE: 1/29/18
> "
> In the end, what I’m thinking is this:
>
> We have an ambari service that runs the notification -> zookeeper
> it reads the ‘registration area’ from zookeeper to get it’s state and what to
> watch
> post 777 when parsers are installed and registered it is trivial to have my
> installer also register the files to watch
>
> the notifications service also has a notification from zookeeper for new
> registrations.
>
> On notify event, the ‘notification node’ has it’s content set to the event
> details and time
> which the parser would pick up…. causing the reload
> "
>
> POC:
> [hdfs-inotify-zookeeper|https://github.com/ottobackwards/hdfs-inotify-zookeeper]
>
> !inotify-ascii.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
