Hi Venkatesh, If you're using the default settings included in the sample configs, it'll expect JSON data in a special format to support passing schemas along with the data. This is turned on by default because it makes it possible to work with a *lot* more connectors and data storage systems (many require schemas!), though it does mean consuming regular JSON data won't work out of the box. You can easily switch this off by changing these lines in the worker config:
key.converter.schemas.enable=true value.converter.schemas.enable=true to be false instead. However, note that this will only work with connectors that can work with "schemaless" data. This wouldn't work for, e.g., writing Avro files in HDFS since they need schema information, but it might work for other formats. This would allow you to consume JSON data from any topic it already existed in. Note that JSON is not the only format you can use. You can also substitute other implementations of the Converter interface. Confluent has implemented an Avro version that works well with our schema registry ( https://github.com/confluentinc/schema-registry/tree/master/avro-converter). The JSON implementation made sense to add as the one included with Kafka simply because it didn't introduce any other dependencies that weren't already in Kafka. It's also possible to write implementations for other formats (e.g. Thrift, Protocol Buffers, Cap'n Proto, MessagePack, and more), but I'm not aware of anyone who has started to tackle those converters yet. -Ewen On Tue, Nov 10, 2015 at 1:23 PM, Venkatesh Rudraraju < venkatengineer...@gmail.com> wrote: > Hi, > > I am trying out the new kakfa connect service. > > version : kafka_2.11-0.9.0.0 > mode : standalone > > I have a conceptual question on the service. > > Can I just start a sink connector which reads from Kafka and writes to say > HDFS ? > From what I have tried, it's expecting a source-connector as well because > the sink-connector is expecting a particular pattern of the message in > kafka-topic. > > Thanks, > Venkat > -- Thanks, Ewen
