[
https://issues.apache.org/jira/browse/NIFI-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485040#comment-15485040
]
ASF GitHub Bot commented on NIFI-1170:
--------------------------------------
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/980#discussion_r78437746
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java
---
@@ -245,58 +490,82 @@ private void recoverState(final ProcessContext
context, final Map<String, String
+ "this indicates that the file has rotated. Will
begin tailing current file from beginning.", new
Object[]{existingTailFile.length(), position});
}
- state = new TailFileState(currentFilename, tailFile, reader,
position, timestamp, checksum, ByteBuffer.allocate(65536));
+ states.get(filePath).setState(new TailFileState(filePath,
tailFile, reader, position, timestamp, length, checksum,
ByteBuffer.allocate(65536)));
} else {
- resetState(currentFilename);
+ resetState(filePath);
}
- getLogger().debug("Recovered state {}", new Object[]{state});
+ getLogger().debug("Recovered state {}", new
Object[]{states.get(filePath).getState()});
}
- private void resetState(final String currentFilename) {
- expectedRecoveryChecksum = null;
- state = new TailFileState(currentFilename, null, null, 0L, 0L,
null, ByteBuffer.allocate(65536));
+ private void resetState(final String filePath) {
+ states.get(filePath).setExpectedRecoveryChecksum(null);
+ states.get(filePath).setState(new TailFileState(filePath, null,
null, 0L, 0L, 0L, null, ByteBuffer.allocate(65536)));
}
@OnStopped
public void cleanup() {
- final TailFileState state = this.state;
- if (state == null) {
+ for (TailFileObject tfo : states.values()) {
+ cleanReader(tfo);
+ final TailFileState state = tfo.getState();
+ tfo.setState(new TailFileState(state.getFilename(),
state.getFile(), null, state.getPosition(), state.getTimestamp(),
state.getLength(), state.getChecksum(), state.getBuffer()));
+ }
+ }
+
+ private void cleanReader(TailFileObject tfo) {
+ if (tfo.getState() == null) {
return;
}
- final FileChannel reader = state.getReader();
+ final FileChannel reader = tfo.getState().getReader();
if (reader == null) {
return;
}
try {
reader.close();
+ getLogger().debug("Closed FileChannel {}", new
Object[]{reader});
} catch (final IOException ioe) {
getLogger().warn("Failed to close file handle during cleanup");
}
-
- getLogger().debug("Closed FileChannel {}", new Object[]{reader});
-
- this.state = new TailFileState(state.getFilename(),
state.getFile(), null, state.getPosition(), state.getTimestamp(),
state.getChecksum(), state.getBuffer());
}
@Override
public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
+ if(isMultiChanging.get()) {
+ long timeSinceLastLookup = new Date().getTime() -
lastLookup.get();
+ if(timeSinceLastLookup >
context.getProperty(LOOKUP_FREQUENCY).asTimePeriod(TimeUnit.MILLISECONDS)) {
+ try {
+ recoverState(context);
+ } catch (IOException e) {
+ getLogger().warn("Exception raised while looking up
for new files", e);
--- End diff --
Good catch.
> TailFile "File to Tail" property should support Wildcards
> ---------------------------------------------------------
>
> Key: NIFI-1170
> URL: https://issues.apache.org/jira/browse/NIFI-1170
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 0.4.0
> Reporter: Andre
>
> Because of challenges around log rotation of high volume syslog and app
> producers, it is customary to logging platform developers to promote file
> variables based file names such as DynaFiles (rsyslog), Macros(syslog-ng)as
> alternatives to getting SIGHUPs being sent to the syslog daemon upon every
> file rotation.
> (To certain extent, used even NiFi's has similar patterns, like for example,
> when one uses Expression Language to set PutHDFS destination file).
> The current TailFile strategy suggests rotation patterns like:
> {code}
> log_folder/app.log
> log_folder/app.log.1
> log_folder/app.log.2
> log_folder/app.log.3
> {code}
> It is possible to fool the system to accept wildcards by simply using a
> strategy like:
> {code}
> log_folder/test1
> log_folder/server1
> log_folder/server2
> log_folder/server3
> {code}
> And configure *Rolling Filename Pattern* to * but it feels like a hack,
> rather than catering for an ever increasingly prevalent use case
> (DynaFile/macros/etc).
> It would be great if instead, TailFile had the ability to use wildcards on
> File to Tail property
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
