In the future, when the ‘filter paths on the name node side for inotify’ lands in hdfs ( there is a jira from the summer that is not making progress ) we can just use the paths to register.
On January 26, 2018 at 08:47:11, Otto Fowler (ottobackwa...@gmail.com) wrote: In the end, what I’m thinking is this: We have an ambari service that runs the notification -> zookeeper it reads the ‘registration area’ from zookeeper to get it’s state and what to watch post 777 when parsers are installed and registered it is trivial to have my installer also register the files to watch the notifications service also has a notification from zookeeper for new registrations. On notify event, the ‘notification node’ has it’s content set to the event details and time which the parser would pick up…. causing the reload ??? profit This would work for the future script parser etc etc. On January 26, 2018 at 08:30:32, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Interesting, so you have an INotify listener to filter events, and then on given changes, propagate a notification to zookeeper, which then triggers the reconfiguration event via the curator client in Metron. I kinda like it given our existing zookeeper methods. Simon On 26 Jan 2018, at 13:27, Otto Fowler <ottobackwa...@gmail.com> wrote: https://github.com/ottobackwards/hdfs-inotify-zookeeper Working on a poc On January 26, 2018 at 07:41:44, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Should we consider using the Inotify interface to trigger reconfiguration, in same way we trigger config changes in curator? We also need to fix caching and lifecycle in the Grok parser to make the zookeeper changes propagate pattern changes while we’re at it. Simon > On 26 Jan 2018, at 03:16, Casey Stella <ceste...@gmail.com> wrote: > > Right now you have to restart the parser topology. > > On Thu, Jan 25, 2018 at 10:15 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> At the moment, when a grok file or something changes in HDFS, how do we >> know? Do we have to restart the parser topology to pick it up? >> Just trying to clarify for myself. >> >> ottO >>
