I am having the same problem except that I am only copying files one at a time and this occurs after I copy the first file. My configs are as follows.
1) First box # Name the compnents of the agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 ###############Describe/configure the source################# a1.sources.r1.type = spooldir a1.sources.r1.spoolDir = /home/desktop/flume_test/ a1.sources.r1.deletePolicy = immediate a1.sources.r1.trackerDir = /home/desktop/.flume/ a1.sources.r1.deserializer.maxLineLength = 10000 # how many events to transfer to the channel at a time a1.sources.r1.batchSize = 1000 a1.sources.r1.channels = c1 ##############describe the sink####################### # Avro sink a1.sinks.k1.type = avro a1.sinks.k1.hostname = box2 a1.sinks.k1.port = 9313 # How many events to take from the channel at a time a1.sinks.k1.batch-size = 1000 # Channel the sink connects to a1.sinks.k1.channel = c1 ################describe the channel################## # use a channel which buffers events in memory a1.channels.c1.type = memory # How many events the channel holds a1.channels.c1.capacity = 10000 # Max number of events to give to sink or take from source a1.channels.c1.transactionCapacity = 1000 a1.channels.c1.byteCapacity = 0 2) Second Box # Name the compnents of the agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 ###############Describe/configure the source################# a1.sources.r1.type = avro a1.sources.r1.bind = box2 a1.sources.r1.port = 9313 a1.sources.r1.channels = c1 ##############describe the sink####################### # HDFS sink a1.sinks.k1.type = hdfs a1.sinks.k1.hdfs.path = /user/hive a1.sinks.k1.hdfs.fileType = DataStream a1.sinks.k1.hdfs.rollSize = 0 a1.sinks.k1.hdfs.rollCount = 5000000 a1.sinks.k1.hdfs.rollInterval = 0 a1.sinks.k1.hdfs.idleTimeout = 30 a1.sinks.k1.hdfs.batchSize = 1000 # Channel the sink connects to a1.sinks.k1.channel = c1 ################describe the channel################## # use a channel which buffers events in memory a1.channels.c1.type = memory # How many events the channel can hold a1.channels.c1.capacity = 10000 # Max number of events the channel can take from source # or give to a sink a1.channels.c1.transactionCapacity = 1000 a1.channels.c1.byteCapacity = 0 any help would be appreciated. Chris
