Are you sure it is not the spooling directory source that is not able to write more data. What makes you think that it is avro sink to source that tis the issue? It just might be that your source is not writing data fast enough or your channel is getting full too fast and the channel is backing off too often?
Thanks, Hari On Monday, November 11, 2013 at 7:16 AM, Christopher Shannon wrote: > Hello, > > I am trying to get a decent throughput rate of transfer for an Avro sink to > Avro Source transfer, but the best I've been able to get is about 300 records > a minute. This thread > (http://mail-archives.apache.org/mod_mbox/flume-user/201309.mbox/%3ccahsgzwvdbdvpzr4-19dsjjtoksv0btqrqmrn5grj87lcmjk...@mail.gmail.com%3e) > on the mailing list suggests that hardware and disk partitioning is the main > consideration to achieving good throughput. > > My setup is simple: > > Agent 1: > Source: SpoolingDirectory > Channel: Memory > Sink: Avro > > Agent 2: > Source: Avro > Channel: Memory > Sink: HDFS > > So, what else needs to be done to optimize the data transfer? > > All the best, > > Chris > > p.s. Anticipating suggestion that I upgrade to Flume 1.4, I work for a big > company that has acquired Flume bundled through another product from an even > bigger company. They will not consider an upgrade at this time. (And given > the very slow throughput I'm seeing, I would think that 1.3 can do a lot > better.) > > >
