Jeremy, I am interested in a JDBC flume sink are you open sourcing it? -Steve
On Tue, Nov 26, 2013 at 8:52 PM, Jeremy Karlson <[email protected]>wrote: > Is there any interest in a generic JDBC sink? > > Over the few days I decided to try and write one. I have something that > requires more testing, but seems to be working. > > Since the config file is how you’d interact with it, here’s a working > example from my source tree: > > a.sinks.k.type=jdbc > a.sinks.k.channel=c > a.sinks.k.driver=com.mysql.jdbc.Driver > a.sinks.k.url=jdbc:mysql://localhost:8889/flume > a.sinks.k.user=username > a.sinks.k.password=password > a.sinks.k.batchSize=100 > a.sinks.k.sql=insert into twitter (body, timestamp) values > (${body:string}, ${header.timestamp:long}) > > The interesting part is the SQL statement. You can put anything you want > in there - it will get converted to a prepared statement on execution. The > Ant-ish tokens get parsed and replaced with parameters at startup. > > The tokens are three part. For example, in: > > ${body:string(UTF-8)} > > The first is a place in the event to get the value from (“body”, > “header.foo”, or “custom”). The second part ("string") is a type > identifier that converts into an appropriate JDBC parameter. The third > part (“UTF-8") is a configuration string for that type, if needed. As for > types, so far I’ve defined: > > body: string (with optional charset encoding), bytearray > header: string, long, int, float, double, date (with mandatory date format > and optional timezone) > > Additionally, if none of those make you happy you can define you own > parameter converters: > > ${custom:com.company.foo.MyConverter(optionaltextconfig)} > > I know there is still improvement to be made, but I’d like to get some > feedback, bug fixes, and maybe get it included before I do a bunch of > useless work. If there is interest, how would you like it for review or > inclusion? > > -- Jeremy > > >
