A large batchSize on sink is not good enough ? -roshan
On Thu, Jul 17, 2014 at 8:43 AM, Mark Bramnik <[email protected]> wrote: > Hi, I'm a newbie in a flume world, and I have a question about the custom > sink. > > I would like to use flume process to gather log files and push them into > the database. > I know that I have to implement the sink because we're interested in very > specific database with a very specific table structure which is not > supported by flume. > > So, I need to create a custom Sink which is well explained in flume > documentation. However I would like to process the events from the channel > in bulk (use bulk insert into db). > > What is the best approach to implement such a code? > I'm not sure how to implement this logic given the fact that the event > taken from the channel can't be put back/we don't have something like > peek() in the channel. > > Thanks a lot in advance and have a nice day :) > > Mark > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
