[
https://issues.apache.org/jira/browse/NIFI-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jasper Knulst reassigned NIFI-6286:
-----------------------------------
Assignee: Jasper Knulst
> Make listHDFS work as INPUT_ALLOWED processor
> ---------------------------------------------
>
> Key: NIFI-6286
> URL: https://issues.apache.org/jira/browse/NIFI-6286
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.9.2
> Reporter: Jasper Knulst
> Assignee: Jasper Knulst
> Priority: Major
> Labels: features
>
> Currently the listHDFS processor has a prop 'Directory' (to start the listing
> from, recursively or not) which only allows 1 static value.
> There are many use cases where you would want to crawl many roots in
> sequence. There are 2 ways to do it.
> # Allow the 'Directory' prop to have multiple comma separated values
> # Refactor listHDFS as an INPUT_ALLOWED processor and make the 'Directory'
> prop take EL to get directory roots from upstream
> Option 1. has serious restrictions since it dictates that other config (like
> recursive, filter type and regex) would still be static and may get very
> complex, non-intuitive and require frequent re-configuration.
> Option 2. is the way to go.
> Some things to consider:
> -The way listHDFS behaves now should be preserved
> -It makes sense to dynamically set 'Directory', 'Recursiveness', 'Regex' and
> 'Filter type' in tandem to be able to detail the way each root directory
> is crawled
> -Switching 'Directory' also requires that not just 1 state is stored but
> states for each directory that ever passed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
