Looks like you'll have three, then :) On Fri, Aug 14, 2015 at 8:34 AM, Steven Jacobs <[email protected]> wrote:
> I would like to as well. > Steven > > On Friday, August 14, 2015, Murtadha Hubail <[email protected]> wrote: > > > I would be happy to review it. > > > > Regards, > > Murtadha > > > > > On Aug 14, 2015, at 10:34 AM, abdullah alamoudi <[email protected] > > <javascript:;>> wrote: > > > > > > Hi, > > > I just want to give a little update about what I have been up to > > recently. > > > I have completed the design and implementation of the > > > FileSystemFeedAdapter. The adapter basically gets a list of directories > > as > > > a parameter and start ingesting records in files that exists within the > > > specified directories. > > > > > > I am using the Java WatchService API to monitor the selected > directories. > > > With this API, the system receives three types of notifications: > > > 1. File Create notification. This notification is sent whenever a new > > file > > > is created or a file has been renamed. > > > 2. File Delete notification. This notification is sent whenever an > > existing > > > file is deleted or a file has been renamed. > > > 3. File Modification notification. This notification is sent whenever > > > changes to an existing file are saved. > > > > > > We would like to avoid reading files which are being written and so we > > > added another parameter that the user is expected to supply, > > > "matching-expression". This expression is matched against File Create > > > notifications and will start immediately ingesting data from the file. > > The > > > user is expected to write their data in a file with a name that doesn't > > > match the expression (i.e., something that ends with .tmp, .staging, > etc) > > > and rename it to something that matches the expression when the file is > > > ready to be read. I think many other systems do the same. > > > > > > The feed stops working if all the watched directories have been removed > > or > > > when the user disconnect the feed. > > > > > > The only remaining piece of this is how we can address feed > continuation > > in > > > case of system failure as right now, I am not storing progress on disk > at > > > all. This however will be added in another iteration. > > > > > > A change request will be submitted for this today and I am thinking > about > > > selecting @Murtadha, @Ian for review. Please, confirm that you have > time > > to > > > do it :-) and thanks in advance :-). I would've picked @Raman but he > > seems > > > very busy these days. > > > > > > If any of this doesn't make sense or if there is a better way to do any > > of > > > this, please let me know. > > > > > > Cheers, > > > Abdullah. > > > > >
