Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/980#discussion_r78954211
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java
---
@@ -117,31 +173,78 @@
.allowableValues(LOCATION_LOCAL, LOCATION_REMOTE)
.defaultValue(LOCATION_LOCAL.getValue())
.build();
+
static final PropertyDescriptor START_POSITION = new
PropertyDescriptor.Builder()
.name("Initial Start Position")
- .description("When the Processor first begins to tail data,
this property specifies where the Processor should begin reading data. Once
data has been ingested from the file, "
+ .description("When the Processor first begins to tail data,
this property specifies where the Processor should begin reading data. Once
data has been ingested from a file, "
+ "the Processor will continue from the last point
from which it has received data.")
.allowableValues(START_BEGINNING_OF_TIME, START_CURRENT_FILE,
START_CURRENT_TIME)
.defaultValue(START_CURRENT_FILE.getValue())
.required(true)
.build();
+ static final PropertyDescriptor RECURSIVE = new
PropertyDescriptor.Builder()
+ .name("tailfile-recursive-lookup")
+ .displayName("Recursive lookup")
+ .description("When using Multiple files mode, this property
defines if files must be listed recursively or not"
+ + " in the base directory.")
+ .allowableValues("true", "false")
+ .defaultValue("true")
+ .required(true)
+ .build();
+
+ static final PropertyDescriptor ROLLING_STRATEGY = new
PropertyDescriptor.Builder()
+ .name("tailfile-rolling-strategy")
+ .displayName("Rolling Strategy")
+ .description("Specifies if the files to tail have a fixed name
or not.")
+ .required(true)
+ .allowableValues(FIXED_NAME, CHANGING_NAME)
+ .defaultValue(FIXED_NAME.getValue())
+ .build();
+
+ static final PropertyDescriptor LOOKUP_FREQUENCY = new
PropertyDescriptor.Builder()
+ .name("tailfile-lookup-frequency")
+ .displayName("Lookup frequency")
+ .description("Only used in Multiple files mode and Changing
name rolling strategy, it specifies the minimum "
+ + "duration the processor will wait before listing
again the files to tail.")
+ .required(false)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 minutes")
--- End diff --
@olegz @joewitt
Just updated the PR with a default of 10 minutes for refresh frequency, and
24 hours for maximum age as a daily rollover pattern seems to be indeed the
most common case. I updated the property descriptions and additional details
page.
I also changed a little bit the approach on the regex matching while
listing files to tail. Previously I was only matching the regular expression
against file name, now I am trying to match the regular expression against the
path. It gives more flexibility to the users. I added an example in the
additional details.
And I squashed my commits.
Let me know your thoughts.
@trixpan, I understand your point but it sounds like an edge case, no? It
would require to tail a very large number of files to get in such a situation.
I'd be inclined to see this as an additional improvement in a separate PR if
you agree.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
