[
https://issues.apache.org/jira/browse/FLUME-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506778#comment-13506778
]
wolfgang hoschek commented on FLUME-1753:
-----------------------------------------
Forgot to mention that the files are not all contained in the same directory so
the --dirname option can't be used.
> Add support for submitting multiple files per avro client command line
> invocation
> ---------------------------------------------------------------------------------
>
> Key: FLUME-1753
> URL: https://issues.apache.org/jira/browse/FLUME-1753
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.3.0
> Reporter: wolfgang hoschek
>
> Some of the use cases we have involve sending a bunch of little files into a
> flume pipe. Would give better perf to send them all with one flume-ng
> avro-client command line invocation, rather than N processes started up
> serially.
> Perhaps have flume be able to read a list of input files from stdin,
> generated by linux find command or HDFS 'find' command or similar. Along the
> following lines:
> find ... |flume-ng avro-client --inputlist=-
> and from a file containing one URL per line:
> flume-ng avro-client --inputlist=<file>
> and from N files:
> flume-ng avro-client —-filename=<file1> —-filename=<file2> ...
> —-filename=<fileN>
> A file can be a local file or an HDFS path URL.
> In our case the headers would be the same for all files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
