Venkatesh, 1. It only works with quotes because the message needs to be parsed as JSON -- a bare string without quotes is not valid JSON. If you're just using a file sink, you can also try the StringConverter, which only supports strings and uses a fixed schema, but is also very easy to use since it has minimal requirements. It's really meant for demonstration purposes more than anything else, but may be helpful just to get up and running. 2. Which JsonParser error? When processing a message fails, we need to be careful about how we handle it. Currently it will not proceed if it can't process a message since for a lot of applications it isn't acceptable to drop messages. By default, we want at least once semantics, with exactly once as long as we don't encounter any crashes or network errors. Manual intervention is currently required in that case.
-Ewen On Tue, Nov 10, 2015 at 8:58 PM, Venkatesh Rudraraju < venkatengineer...@gmail.com> wrote: > Hi Ewen, > > Thanks for the explanation. with your suggested setting, I was able to > start just a sink connector like below : > > >* bin/connect-standalone.sh config/connect-standalone.properties > config/connect-file-sink.properties* > > But I have a couple of issues yet, > 1) Since I am only testing a simple file sink connector, I am manually > producing some messages to the 'connect-test' kafka topic, where the > sink-Task is reading from. And it works only if the message is within > double-quotes. > 2) Once I hit the above JsonParser error on the SinkTask, the connector is > hung, doesn't take any more messages even proper ones. > > > On Tue, Nov 10, 2015 at 1:59 PM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > Hi Venkatesh, > > > > If you're using the default settings included in the sample configs, > it'll > > expect JSON data in a special format to support passing schemas along > with > > the data. This is turned on by default because it makes it possible to > work > > with a *lot* more connectors and data storage systems (many require > > schemas!), though it does mean consuming regular JSON data won't work out > > of the box. You can easily switch this off by changing these lines in the > > worker config: > > > > key.converter.schemas.enable=true > > value.converter.schemas.enable=true > > > > to be false instead. However, note that this will only work with > connectors > > that can work with "schemaless" data. This wouldn't work for, e.g., > writing > > Avro files in HDFS since they need schema information, but it might work > > for other formats. This would allow you to consume JSON data from any > topic > > it already existed in. > > > > Note that JSON is not the only format you can use. You can also > substitute > > other implementations of the Converter interface. Confluent has > implemented > > an Avro version that works well with our schema registry ( > > > https://github.com/confluentinc/schema-registry/tree/master/avro-converter > > ). > > The JSON implementation made sense to add as the one included with Kafka > > simply because it didn't introduce any other dependencies that weren't > > already in Kafka. It's also possible to write implementations for other > > formats (e.g. Thrift, Protocol Buffers, Cap'n Proto, MessagePack, and > > more), but I'm not aware of anyone who has started to tackle those > > converters yet. > > > > -Ewen > > > > On Tue, Nov 10, 2015 at 1:23 PM, Venkatesh Rudraraju < > > venkatengineer...@gmail.com> wrote: > > > > > Hi, > > > > > > I am trying out the new kakfa connect service. > > > > > > version : kafka_2.11-0.9.0.0 > > > mode : standalone > > > > > > I have a conceptual question on the service. > > > > > > Can I just start a sink connector which reads from Kafka and writes to > > say > > > HDFS ? > > > From what I have tried, it's expecting a source-connector as well > because > > > the sink-connector is expecting a particular pattern of the message in > > > kafka-topic. > > > > > > Thanks, > > > Venkat > > > > > > > > > > > -- > > Thanks, > > Ewen > > > > > > -- > Victory awaits him who has everything in order--luck, people call it. > -- Thanks, Ewen
