> On Feb. 27, 2015, 6:22 a.m., Hari Shreedharan wrote: > > I didn't do a deep dive, but I have one question - if you look at the HBase > > sink or HDFS sink, a pluggable serializer allows the user to convert Flume > > events into an HBase or HDFS friendly form. Is something like this not > > required in case of this sink? Looks like the current implementation > > depends on data in the headers to decide where to put the data. > > Configuration is a better way to achieve this, so I think having a > > serializer (+ a good default one which possibly looks at the headers?) > > would make sense, no? > > Ashish Paliwal wrote: > Agree for the need of Serializer here. Without it, we might be storing > the event as a blob in C* rt? Let me spend some more time with review
Alright! I would make the current behaviour the default serializer. Storing the event as a blob without any extra information (e.g. primary key) will probably not be possible. The approach we are following for default Cassandra/JDBC/etc serializers is trying to automatically map event headers to target schema, i.e. any event header with a name matching a field in the schema will be converted to an appropriate data type and inserted. This works for most of our use cases (we use interceptors in the source to transform each event an populate headers) and for corner cases we use a serializer with template queries, allowing the user to specify an arbitrary statement with some placeholders for headers or body. I'll upload a new version ASAP with the serializers and some additional functionality (inserting to multiple tables at once). - Santiago ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30675/#review74452 ----------------------------------------------------------- On Feb. 5, 2015, 5:19 p.m., Santiago Mola wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30675/ > ----------------------------------------------------------- > > (Updated Feb. 5, 2015, 5:19 p.m.) > > > Review request for Flume. > > > Bugs: FLUME-2616 > https://issues.apache.org/jira/browse/FLUME-2616 > > > Repository: flume-git > > > Description > ------- > > Add Cassandra sink > > > Diffs > ----- > > flume-ng-doc/sphinx/FlumeUserGuide.rst 7a1dfce > flume-ng-sinks/flume-cassandra-sink/pom.xml PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraSink.java > PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraTable.java > PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/main/java/org/apache/flume/sink/cassandra/CassandraUtils.java > PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraDataTypesIT.java > PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraSinkIT.java > PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/CassandraTestHelper.java > PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraSink.java > PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraTable.java > PRE-CREATION > > flume-ng-sinks/flume-cassandra-sink/src/test/java/org/apache/flume/sink/cassandra/TestCassandraUtils.java > PRE-CREATION > flume-ng-sinks/flume-cassandra-sink/src/test/resources/log4j.properties > PRE-CREATION > flume-ng-sinks/pom.xml 4bac019 > pom.xml 1350fa4 > > Diff: https://reviews.apache.org/r/30675/diff/ > > > Testing > ------- > > Extensive unit tests and integration tests. > > > Thanks, > > Santiago Mola > >
