What leads you to that conclusion about FC? (I am curious in case there is something I am unaware of.) This is where a Put ends up being written and there is no flush until a commit.
https://github.com/apache/flume/blob/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java#L165 Brock On Wed, Jul 11, 2012 at 7:12 AM, Patrick Wendell <[email protected]> wrote: > Hi All, > > Most streaming systems have built-in support for batching since it > often offers major performance benefits in terms of throughput. > > I'm a little confused about the state of batching in Flume today. It > looks like a ChannelProcessor can process a batch of events within one > transaction, but internally this just calls Channel.put() several > times. > > As far as I can tell, both of the durable channels (JDBC and File) > actually flush to disk in some fashion whenever there is a doPut(). It > seems to me like it makes sense to buffer all of those puts in memory > and only flush them once per transaction. Otherwise, isn't the benefit > of batching put()'s within a transaction lost? > > I think I might be missing something here, any pointers are appreciated. > > - Patrick > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
