Re: Questions about .9 consumer API

Hi Mohit: In general new consumers will abstract developers from any network failures. More specifically. 1) consumers will automatically try to re-fetch the messages if the previous fetch has failed. 2) consumers will remember the currently fetch positions after each successful fetch, and can

Re: future of Camus?

Take a look at secor: https://github.com/pinterest/secor Secor is a no-frill kafka->HDFS/Ingesting tool, doesn't depend on any underlying systems such as Hadoop, it only uses Kafka high level consumer to balance the work loads. Very easy to understand and manage. It's probably the 2nd most

future of Camus?

Hello all, We're looking at options for getting data from Kafka onto HDFS and Camus looks like the natural choice for this. It's also evident that LinkedIn who originally created Camus are taking things in a different direction and are advising people to use their Gobblin ETL framework

Re: future of Camus?

Another alternative is to checkout Kaboom ‎ https://github.com/blackberry/KaBoom ‎It uses a pared down kafka consumer library to pull data from Kafka and write it to defined (and somewhat dynamic) hdfs paths in a custom (and changeable) avro schema we call boom. It uses kerberos for

kafka 0.8 consumer polling topic

Hi all General question does the current kafka consumer needs to be written in java threads in order to poll the topic continuously and how should it be written any ideas please? Thanks Regards Surender Kudumula Big Data Consultant - EMEA Analytics & Data Management

Re: future of Camus?

Hi Adrian, Another alternative approach is to use Kafka's own Copycat framework for data ingressing / egressing. It will be released in our 0.9.0 version expected in Nov. Under Copycat users can write different "connector" instantiated for different source / sink systems, while for your case

Re: future of Camus?

We are using Apache Flume as a router to consume data from Kafka and push to HDFS. With Flume 1.6, Kafka Channel, Source and Sink are available out of the box. Here is the blog post from Cloudera http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/

Re: future of Camus?

Very useful information for us. Thanks Guozhang. On Oct 22, 2015 2:02 PM, "Guozhang Wang" wrote: > Hi Adrian, > > Another alternative approach is to use Kafka's own Copycat framework for > data ingressing / egressing. It will be released in our 0.9.0 version > expected in

Re: kafka 0.8 consumer polling topic

You can find the Java doc with some examples under "KafkaConsumer" here: http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ Guozhang On Thu, Oct 22, 2015 at 1:28 PM, Kudumula, Surender < surender.kudum...@hpe.com> wrote: > Hi all > > General question does the current kafka

Re: Questions about .9 consumer API

It's in this link. Most of the examples have some kind of error handling http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ On Thu, Oct 22, 2015 at 7:45 PM, Guozhang Wang wrote: > Could you point me to the exact examples that indicate user error

Re: Questions about .9 consumer API

Could you point me to the exact examples that indicate user error handling? Guozhang On Thu, Oct 22, 2015 at 5:43 PM, Mohit Anchlia wrote: > The examples in the javadoc seems to imply that developers need to manage > all of the aspects around failures. Those examples

Questions about .9 consumer API

It looks like the new consumer API expects developers to manage the failures? Or is there some other API that can abstract the failures, primarily: 1) Automatically resent failed messages because of network issue or some other issue between the broker and the consumer 2) Ability to acknowledge