Re: Kafka to Hadoop HDFS

[max square]

This presentation from a recent Kafka meetup in NYC describes different
approaches.
http://www.slideshare.net/gwenshap/kafka-hadoop-for-nyc-kafka-meetup?ref=http://ingest.tips/2014/10/16/notes-from-kafka-meetup/

It´s companion blog post is this:
http://ingest.tips/2014/10/16/notes-from-kafka-meetup/

Cheers

Max

On Sat, Mar 7, 2015 at 12:18 AM, Daniel Moreno d...@max2.com wrote:

 You can also try the approach described here
 http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/

 Kind Regards,

 Daniel


 On Mar 6, 2015, at 23:20, Lin Ma lin...@gmail.commailto:lin...@gmail.com
 wrote:

 Hi Kafka masters,

 Wondering if any open source solutions, to transfer message received from
 Kakfa to Hadoop HDFS? Thanks.

 regards,
 Lin

topics still showing up using list command after deletion

[max square]

Hi All,

I am using Kafka version 0.8.2.1. Like mentioned in the documentation, I
enabled the delete.kafka.topic property in the config, restarted the
brokers and issued the delete command. Then, I tried listing the topics and
the topics that I deleted still shows up in the list. However, if I try the
delete command again for those topics, it says that those topics don't
exist. Any help in this regard would be appreciated.

Thank you,
Max

Attaching the commands that I used :-
bin/kafka-topics.sh --zookeeper x.x.x.x:2181/chroot --delete --topic test
bin/kafka-topics.sh --list --zookeeper x.x.x.x:2181

Re: Trying to get kafka data to Hadoop

[max square]

Thunder,

thanks for your reply. The hadoop job is now correctly configured (the
client was not getting the correct jars), however I am getting Avro
formatting exceptions due to the format the schema-repo server follows. I
think I will do something similar and create our own branch that uses the
schema repo. Any gotchas you can advice on?

Thanks!

Max

On Wed, Mar 4, 2015 at 9:24 PM, Thunder Stumpges tstump...@ntent.com
wrote:

 What branch of camus are you using? We have our own fork that we updated
 the camus dependency from the avro snapshot of the REST Schema Repository
 to the new official one you mention in github.com/schema-repo. I was
 not aware of a branch on the main linked-in camus repo that has this.

 That being said, we are doing essentially this same thing however we are
 using a single shaded uber-jar. I believe the maven project builds this
 automatically doesnt it?

 I'll take a look at the details of how we are invoking this on our site
 and get back to you.

 Cheers,
 Thunder


 -Original Message-
 From: max square [max2subscr...@gmail.com]
 Received: Wednesday, 04 Mar 2015, 5:38PM
 To: users@kafka.apache.org [users@kafka.apache.org]
 Subject: Trying to get kafka data to Hadoop

 Hi all,

 I have browsed through different conversations around Camus, and bring this
 as a kinda Kafka question. I know is not the most orthodox, but if someone
 has some thoughts I'd appreciate ir.

 That said, I am trying to set up Camus, using a 3 node Kafka cluster
 0.8.2.1, using a project that is trying to build Avro Schema-Repo
 https://github.com/schema-repo/schema-repo. All of the Avro schemas for
 the topics are published correctly. I am building Camus and using:

 hadoop jar camus-example-0.1.0-SNAPSHOT-shaded.jar com.linkedin.camus.etl.
 kafka.CamusJob -libjars $CAMUS_LIBJARS  -D mapreduce.job.user.classpath.
 first=true -P config.properties

 As the command to start the job, where I have set up an environment
 variable that holds all the libjars that the mvn package command generates.

 I have also set the following properties to configure the job:
 camus.message.decoder.class=com.linkedin.camus.etl.kafka.coders.
 LatestSchemaKafkaAvroMessageDecoder

 kafka.message.coder.schema.registry.class=com.linkedin.camus.schemaregistry.
 AvroRestSchemaRegistry

 etl.schema.registry.url=http://10.0.14.25:2876/schema-repo/

 When I execute the job I get an Exception indicating the
 AvroRestSchemaRegistry class can't be found (I've double checked it's part
 of the libjars). I wanted to ask if this is the correct way to set up this
 integration, and if anyone has pointers on why the job is not finding the
 class AvroRestSchemaRegistry

 Thanks in advance for the help!

 Max

 Follows the complete stack trace:

 [CamusJob] - failed to create decoder

 com.linkedin.camus.coders.MessageDecoderException:
 com.linkedin.camus.coders
 .MessageDecoderException:java.lang.ClassNotFoundException: com.linkedin.
 camus.schemaregistry.AvroRestSchemaRegistry
at com.linkedin.camus.etl.kafka.coders.MessageDecoderFactory.
 createMessageDecoder(MessageDecoderFactory.java:29)

   at
 com.linkedin.camus.etl.kafka.mapred.EtlInputFormat.createMessageDecoder
 (EtlInputFormat.java:391)

at com.linkedin.camus.etl.kafka.mapred.EtlInputFormat.getSplits(
 EtlInputFormat.java:256)

at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:
 1107)

at
 org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1124
 )

  at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:178)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1023)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(
 UserGroupInformation.java:1642)

at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.
 java:976)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:582)

at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:335)

at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:563)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

at com.linkedin.camus.etl.kafka.CamusJob.main(CamusJob.java:518)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodessorImpl.
 java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(
 DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.