[
https://issues.apache.org/jira/browse/FLUME-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415262#comment-15415262
]
Andrea Rota commented on FLUME-2911:
------------------------------------
Hi [~bessbd], sure!
This is the specific case we dealt: we had a folder (D:/flume/spooldir) where
we didn't know in advance what kind of files where written by several
processes. We just wanted to transport the one ending with .READY extension.
remote.sources.dirSource.type = spooldir
remote.sources.dirSource.channels = fileChannel
remote.sources.dirSource.spoolDir = D:/flume/spooldir
remote.sources.dirSource.includePattern = ^.*\.READY$
remote.sources.dirSource.fileHeader = true
remote.sources.dirSource.deletePolicy = immediate
Declaring this kind of condition with ignorePattern requires a negative regex,
which is very tricky and needs to be updated when a new type of file appears in
the folder.
When both the ignorePattern and includePattern matches, the code stays on the
safe side and ignore the files. Do you want me to edit the documentation?
Cheers
> Add includePattern option in SpoolDirectorySource configuration
> ---------------------------------------------------------------
>
> Key: FLUME-2911
> URL: https://issues.apache.org/jira/browse/FLUME-2911
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: notrack, v1.6.0, v1.7.0
> Reporter: Andrea Rota
> Labels: features
> Attachments: FLUME-2911.patch
>
>
> Current implementation of SpoolDirectorySource does not allow users to
> specify a regex pattern to select which files should be monitored. Instead,
> the current implementation allows users to specify which should *not*
> monitored, via the ignorePattern parameter.
> I implemented the feature, allowing users to specify the include pattern as
> {{a1.sources.src-1.includePattern=^foo.*$}} (includes all the files that
> starts in "foo").
> By default, the includePattern regex is set to {{^.*$}} (all files). Include
> and exclude patterns can be used at same time and the results are combined.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
