I take it your service would just be a thin daemon along the lines of the PoC you linked, which makes a lot of sense, delegating the actual notification to the zookeeper bits we already have.
That makes sense to me. One other question would be around the availability of that service (which is not exactly critical, but would be nice to be able to run HA). As far as I can see it’s not likely to be stateful, and as long as there is some sort of de-dupe you could have two or more running. Is that worth chewing, or do we just need one running and accept occasional outages of the rarely firing non-critical service? Simon > On 31 Jan 2018, at 17:24, Otto Fowler <ottobackwa...@gmail.com> wrote: > > No, > > I would propose a new Ambari Service, the did the notify->zookeeper stuff. > Did you not see my awesome ascii art diagram? > > > > > On January 31, 2018 at 11:51:51, Casey Stella (ceste...@gmail.com) wrote: > > Well, it'll be one listener per worker and if you have a lot of workers, > it's going to be a bad time probably. > > On Wed, Jan 31, 2018 at 11:50 AM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> I don’t think the Unstable means the implementation will crash. I think >> it means >> it is a newish-api, and there should be 10000 listeners. >> >> Having 1 listener shouldn’t be an issue. >> >> >> >> On January 31, 2018 at 11:45:54, Casey Stella (ceste...@gmail.com) wrote: >> >> Hmm, I have heard this feedback before. Perhaps a more low-key approach >> would be either a static timer that checked or a timer bolt that sent a >> periodic timer and the parser bolt reconfigured the parser (or indeed we >> added a Reloadable interface with a 'reload' method). We could be smart >> also and only set up the topology with the timer bolt if the parser >> actually implemented the Reloadable interface. Just some thoughts that >> might be easy and avoid instability. >> >> On Tue, Jan 30, 2018 at 3:42 PM, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> It is still @unstable, but the jiras : >>> https://issues.apache.org/jira/browse/HDFS-8940?jql= >>> project%20%3D%20HDFS%20AND%20status%20in%20(Open%2C%20% >>> 22In%20Progress%22)%20AND%20text%20~%20%22INotify%22 >>> that I see are stall from over the summer. >>> >>> They also seem geared to scale or changing the filter object not the api. >>> >>> >>> >>> On January 30, 2018 at 14:19:56, JJ Meyer (jjmey...@gmail.com) wrote: >>> >>> Hello all, >>> >>> I had created a NiFi processor a long time back that used the inotify >> API. >>> One thing I noticed while working with it is that it is marked with the >>> `Unstable` annotation. It may be worth checking if anymore work is going >> on >>> with it and if it will impact this (if it hasn't already been looked >> into). >>> >>> Thanks, >>> JJ >>> >>> On Mon, Jan 29, 2018 at 7:27 AM, Otto Fowler <ottobackwa...@gmail.com> >>> wrote: >>> >>>> I have updated the jira as well >>>> >>>> >>>> On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com) >>>> wrote: >>>> >>>> https://github.com/ottobackwards/hdfs-inotify-zookeeper >>>> >>> >> >>
