I would like to as well. Steven On Friday, August 14, 2015, Murtadha Hubail <[email protected]> wrote:
> I would be happy to review it. > > Regards, > Murtadha > > > On Aug 14, 2015, at 10:34 AM, abdullah alamoudi <[email protected] > <javascript:;>> wrote: > > > > Hi, > > I just want to give a little update about what I have been up to > recently. > > I have completed the design and implementation of the > > FileSystemFeedAdapter. The adapter basically gets a list of directories > as > > a parameter and start ingesting records in files that exists within the > > specified directories. > > > > I am using the Java WatchService API to monitor the selected directories. > > With this API, the system receives three types of notifications: > > 1. File Create notification. This notification is sent whenever a new > file > > is created or a file has been renamed. > > 2. File Delete notification. This notification is sent whenever an > existing > > file is deleted or a file has been renamed. > > 3. File Modification notification. This notification is sent whenever > > changes to an existing file are saved. > > > > We would like to avoid reading files which are being written and so we > > added another parameter that the user is expected to supply, > > "matching-expression". This expression is matched against File Create > > notifications and will start immediately ingesting data from the file. > The > > user is expected to write their data in a file with a name that doesn't > > match the expression (i.e., something that ends with .tmp, .staging, etc) > > and rename it to something that matches the expression when the file is > > ready to be read. I think many other systems do the same. > > > > The feed stops working if all the watched directories have been removed > or > > when the user disconnect the feed. > > > > The only remaining piece of this is how we can address feed continuation > in > > case of system failure as right now, I am not storing progress on disk at > > all. This however will be added in another iteration. > > > > A change request will be submitted for this today and I am thinking about > > selecting @Murtadha, @Ian for review. Please, confirm that you have time > to > > do it :-) and thanks in advance :-). I would've picked @Raman but he > seems > > very busy these days. > > > > If any of this doesn't make sense or if there is a better way to do any > of > > this, please let me know. > > > > Cheers, > > Abdullah. > >
