I would be happy to review it. Regards, Murtadha
> On Aug 14, 2015, at 10:34 AM, abdullah alamoudi <[email protected]> wrote: > > Hi, > I just want to give a little update about what I have been up to recently. > I have completed the design and implementation of the > FileSystemFeedAdapter. The adapter basically gets a list of directories as > a parameter and start ingesting records in files that exists within the > specified directories. > > I am using the Java WatchService API to monitor the selected directories. > With this API, the system receives three types of notifications: > 1. File Create notification. This notification is sent whenever a new file > is created or a file has been renamed. > 2. File Delete notification. This notification is sent whenever an existing > file is deleted or a file has been renamed. > 3. File Modification notification. This notification is sent whenever > changes to an existing file are saved. > > We would like to avoid reading files which are being written and so we > added another parameter that the user is expected to supply, > "matching-expression". This expression is matched against File Create > notifications and will start immediately ingesting data from the file. The > user is expected to write their data in a file with a name that doesn't > match the expression (i.e., something that ends with .tmp, .staging, etc) > and rename it to something that matches the expression when the file is > ready to be read. I think many other systems do the same. > > The feed stops working if all the watched directories have been removed or > when the user disconnect the feed. > > The only remaining piece of this is how we can address feed continuation in > case of system failure as right now, I am not storing progress on disk at > all. This however will be added in another iteration. > > A change request will be submitted for this today and I am thinking about > selecting @Murtadha, @Ian for review. Please, confirm that you have time to > do it :-) and thanks in advance :-). I would've picked @Raman but he seems > very busy these days. > > If any of this doesn't make sense or if there is a better way to do any of > this, please let me know. > > Cheers, > Abdullah.
