Hi, A source can write to multiple channels but a sink can only read from a single channel. So in this case, you probably just want one channel and then have both sources write to that single channel. Notice that for sources it's ".channels = " while for sinks it's ".channel = "
Cheers, Brock On Tue, Sep 25, 2012 at 8:06 AM, Cochran, David M (Contractor) <[email protected]> wrote: > > I'm getting closer, but still doing something wrong.. Do most folks have > this hard a time wrapping their heads around the configs or is it just me? > > -Dave > > > host#1 - tailing 2 source log files... send to host#2 over one channel to > sink FILE_ROLL to be written out. > > > HOST#1 Results- > 25 Sep 2012 07:40:51,289 WARN [conf-file-poller-0] > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks:678) > - Configuration for : avroSink has errors, and will be removed: > org.apache.flume.conf.ConfigurationException: Channel fileChannel > fileChannel1 not in active set. > > > HOST#2 Results- > 25 Sep 2012 06:45:11,568 WARN [conf-file-poller-0] > (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks:678) > - Configuration for : filesink1 has errors, and will be removed: > org.apache.flume.conf.ConfigurationException: Channel fileChannel1 not in > active set. > > > Config for each host follow: > > > HOST#1- > node105.sources = tailsource tailsource1 > node105.channels = fileChannel fileChannel1 > node105.sinks = avroSink > > node105.sources.tailsource.type = exec > node105.sources.tailsource.command = tail -F /home/XXXX/test.log > > node105.sources.tailsource1.type = exec > node105.sources.tailsource1.command = tail -F /home/XXXX/test1.log > > node105.sources.tailsource.channels = fileChannel > node105.sources.tailsource1.channels = fileChannel1 > > ## Sink sends avro messages to xxxx on port 9432 > node105.sinks.avroSink.type = avro > node105.sinks.avroSink.channel = fileChannel fileChannel1 > node105.sinks.avroSink.hostname = 192.168.1.128 > node105.sinks.avroSink.port = 9432 > > node105.channels.fileChannel.type = file > node105.channels.fileChannel.checkpointDir = /tmp/flume/checkpoints > node105.channels.fileChannel.dataDirs = /tmp/flume/data2 > node105.channels.fileChannel.capacity = 10000 > node105.channels.fileChannel.checkpointInterval = 3000 > node105.channels.fileChannel.maxFileSize = 5242880 > > node105.channels.fileChannel1.type = file > node105.channels.fileChannel1.checkpointDir = /tmp/flume/checkpoints1 > node105.channels.fileChannel1.dataDirs = /tmp/flume/data22 > node105.channels.fileChannel1.capacity = 10000 > node105.channels.fileChannel1.checkpointInterval = 3000 > node105.channels.fileChannel1.maxFileSize = 5242880 > > > > > HOST#2 - > node102.sources = avroSource > node102.channels = fileChannel FileChannel1 > node102.sinks = filesink filesink1 > > ## Source listens for avro messages on port 9432 on all ips > node102.sources.avroSource.type = avro > node102.sources.avroSource.channels = fileChannel fileChannel1 > node102.sources.avroSource.bind = 0.0.0.0 > node102.sources.avroSource.port = 9432 > > node102.sinks.filesink.type = FILE_ROLL > node102.sinks.filesink.batchSize = 1000 > node102.sinks.filesink.channel = fileChannel > node102.sinks.filesink.sink.directory = /logs/rhel5/ > node102.sinks.filesink.sink.rollInterval = 86400 > node102.sinks.filesink.sink.serializer = TEXT > > node102.channels.fileChannel.type = file > node102.channels.fileChannel.checkpointDir = /tmp/flume/checkpoints > node102.channels.fileChannel.dataDirs = /tmp/flume/data1 > node102.channels.fileChannel.capacity = 5000 > node102.channels.fileChannel.checkpointInterval = 45000 > node102.channels.fileChannel.maxFileSize = 5242880 > > node102.sinks.filesink1.type = FILE_ROLL > node102.sinks.filesink1.batchSize = 1000 > node102.sinks.filesink1.channel = fileChannel1 > node102.sinks.filesink1.sink.directory = /logs/rhel51/ > node102.sinks.filesink1.sink.rollInterval = 86400 > node102.sinks.filesink1.sink.serializer = TEXT > > node102.channels.fileChannel1.type = file > node102.channels.fileChannel1.checkpointDir = /tmp/flume/checkpoints1 > node102.channels.fileChannel1.dataDirs = /tmp/flume/data2 > node102.channels.fileChannel1.capacity = 5000 > node102.channels.fileChannel1.checkpointInterval = 45000 > node102.channels.fileChannel1.maxFileSize = 5242880 > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
