What if the changes where ‘on top of’ some base set of properties, like directory? Like a filter, where if present from the incoming file will have the LIST* list only things that match a name or attribute?
On March 27, 2018 at 00:08:41, Joe Witt (joe.w...@gmail.com) wrote: Scott This idea has come up a couple of times and there is definitely something intriguing to it. Where I think this idea stalls out though is in implementation. While I agree that the other List* processors might similarly benefit lets focus on ListFile. Today you tell ListFile what directory to start looking for files in. It goes off scanning that directory for hits and stores state about what it has already searched/seen. And it is important to keep track of how much it has already scanned because at times the search directory can be massive (100,000s of thousands or more files and directories to scan for example). In the proposed model the directory to be scanned could be provided dynamically by looking at an attribute of an incoming flowfile (or other criteria can be provided - not just the directory to scan). In this case the ListFile processor goes on scanning against that now. What about the previous directory (or directories) it was told to scan? Does it still track those too? What if it starts scanning the newly provided directory, hasn't finished pulling all the data or new data is continually arriving, and it is told to switch to another directory. I think if those questions can get solid answers and someone invests time in creating a PR then this could be pretty powerful. Would be good to see a written description of the use case(s) for this too. Thanks Joe On Mon, Mar 26, 2018 at 11:58 PM, scott <tcots8...@gmail.com> wrote: > Hello Devs, > > I would like to request a feature to a major processor, ListSFTP. But before > I do down the official road, I wanted to ask if anyone thought it was a > terrible idea or impossible, etc. The request is to add support for an > incoming relationship to the ListSFTP processor specifically, but I could > see it added to many of the commonly used head processes, such as ListFile. > I would envision functionality more like InvokeHTTP or ExecuteSQL, where an > incoming flow file could initiate the action, and the attributes in the > incoming flow file could be used to configure the processor actions. It's > the configuration aspect that most appeals to me, because it opens it up to > being centrally or dynamically configured. > > Thanks, > > Scott >
