Yes, though it's still awaiting some updates after some renaming and API modifications that happened in Kafka recently.
-Ewen On Thu, Nov 12, 2015 at 9:10 AM, Venkatesh Rudraraju < venkatengineer...@gmail.com> wrote: > Ewen, > > How do I use a HDFSSinkConnector. I see the sink as part of a confluent > project ( > > https://github.com/confluentinc/copycat-hdfs/blob/master/src/main/java/io/confluent/copycat/hdfs/HdfsSinkConnector.java > ). > Does it mean that I build this project and add the jar to kafka libs ? > > > > > On Tue, Nov 10, 2015 at 9:35 PM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > Venkatesh, > > > > 1. It only works with quotes because the message needs to be parsed as > JSON > > -- a bare string without quotes is not valid JSON. If you're just using a > > file sink, you can also try the StringConverter, which only supports > > strings and uses a fixed schema, but is also very easy to use since it > has > > minimal requirements. It's really meant for demonstration purposes more > > than anything else, but may be helpful just to get up and running. > > 2. Which JsonParser error? When processing a message fails, we need to be > > careful about how we handle it. Currently it will not proceed if it can't > > process a message since for a lot of applications it isn't acceptable to > > drop messages. By default, we want at least once semantics, with exactly > > once as long as we don't encounter any crashes or network errors. Manual > > intervention is currently required in that case. > > > > -Ewen > > > > On Tue, Nov 10, 2015 at 8:58 PM, Venkatesh Rudraraju < > > venkatengineer...@gmail.com> wrote: > > > > > Hi Ewen, > > > > > > Thanks for the explanation. with your suggested setting, I was able to > > > start just a sink connector like below : > > > > > > >* bin/connect-standalone.sh config/connect-standalone.properties > > > config/connect-file-sink.properties* > > > > > > But I have a couple of issues yet, > > > 1) Since I am only testing a simple file sink connector, I am manually > > > producing some messages to the 'connect-test' kafka topic, where the > > > sink-Task is reading from. And it works only if the message is within > > > double-quotes. > > > 2) Once I hit the above JsonParser error on the SinkTask, the connector > > is > > > hung, doesn't take any more messages even proper ones. > > > > > > > > > On Tue, Nov 10, 2015 at 1:59 PM, Ewen Cheslack-Postava < > > e...@confluent.io> > > > wrote: > > > > > > > Hi Venkatesh, > > > > > > > > If you're using the default settings included in the sample configs, > > > it'll > > > > expect JSON data in a special format to support passing schemas along > > > with > > > > the data. This is turned on by default because it makes it possible > to > > > work > > > > with a *lot* more connectors and data storage systems (many require > > > > schemas!), though it does mean consuming regular JSON data won't work > > out > > > > of the box. You can easily switch this off by changing these lines in > > the > > > > worker config: > > > > > > > > key.converter.schemas.enable=true > > > > value.converter.schemas.enable=true > > > > > > > > to be false instead. However, note that this will only work with > > > connectors > > > > that can work with "schemaless" data. This wouldn't work for, e.g., > > > writing > > > > Avro files in HDFS since they need schema information, but it might > > work > > > > for other formats. This would allow you to consume JSON data from any > > > topic > > > > it already existed in. > > > > > > > > Note that JSON is not the only format you can use. You can also > > > substitute > > > > other implementations of the Converter interface. Confluent has > > > implemented > > > > an Avro version that works well with our schema registry ( > > > > > > > > > > https://github.com/confluentinc/schema-registry/tree/master/avro-converter > > > > ). > > > > The JSON implementation made sense to add as the one included with > > Kafka > > > > simply because it didn't introduce any other dependencies that > weren't > > > > already in Kafka. It's also possible to write implementations for > other > > > > formats (e.g. Thrift, Protocol Buffers, Cap'n Proto, MessagePack, and > > > > more), but I'm not aware of anyone who has started to tackle those > > > > converters yet. > > > > > > > > -Ewen > > > > > > > > On Tue, Nov 10, 2015 at 1:23 PM, Venkatesh Rudraraju < > > > > venkatengineer...@gmail.com> wrote: > > > > > > > > > Hi, > > > > > > > > > > I am trying out the new kakfa connect service. > > > > > > > > > > version : kafka_2.11-0.9.0.0 > > > > > mode : standalone > > > > > > > > > > I have a conceptual question on the service. > > > > > > > > > > Can I just start a sink connector which reads from Kafka and writes > > to > > > > say > > > > > HDFS ? > > > > > From what I have tried, it's expecting a source-connector as well > > > because > > > > > the sink-connector is expecting a particular pattern of the message > > in > > > > > kafka-topic. > > > > > > > > > > Thanks, > > > > > Venkat > > > > > > > > > > > > > > > > > > > > > -- > > > > Thanks, > > > > Ewen > > > > > > > > > > > > > > > > -- > > > Victory awaits him who has everything in order--luck, people call it. > > > > > > > > > > > -- > > Thanks, > > Ewen > > > > > > -- > Victory awaits him who has everything in order--luck, people call it. > -- Thanks, Ewen
