RE: storm-kafka integration duplication of messages
Are you using all grouping for the deserialization bolt too? My original point is that you should not be using this grouping anywhere in your topology. It will cause the duplication you are seeing unless your parallelism is set to one for your bolts. See the section on groupings here https://storm.incubator.apache.org/documentation/Concepts.html and this page about storm parallelism http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/. Also maybe it would help if you share how you are building your topology. -Nathan On Jul 7, 2014 8:06 PM, "Max Evers" wrote: > The AllGrouping was intended to duplicate the messages to HDFS and the > outbound kafka, what were are seeing is the duplicaiton of messages from > the kafkaspout, so the same message is seen multiple times in hdfs AND in > the outbound kafka, as well as the logs of the intermediate bolt handling > the deserialization. Presently there is not much more going on in storm > that the deserialization. I wanted to get the skeleton of the flow working > before I introduced new complications inside of storm. >
RE: storm-kafka integration duplication of messages
The AllGrouping was intended to duplicate the messages to HDFS and the outbound kafka, what were are seeing is the duplicaiton of messages from the kafkaspout, so the same message is seen multiple times in hdfs AND in the outbound kafka, as well as the logs of the intermediate bolt handling the deserialization. Presently there is not much more going on in storm that the deserialization. I wanted to get the skeleton of the flow working before I introduced new complications inside of storm.
Re: FW: storm-kafka integration duplication of messages
Your data is replicated because of all grouping. I assume you have more than kafka and hdfs bolt. If you send via fields grouping (or shuffle grouping, or local or shuffle grouping), and both are subscribed to your deserializer, then one task in hdfs bolt and one task in kafka bolt will get the tuple. The distribution is based on which grouping you use. Since you subscribe using all grouping, then all of the tasks in hdfs bolt and all of the tasks in kafka bolt will get your tuple. On Mon, Jul 7, 2014 at 2:56 PM, Max Evers wrote: > > From: Evers, Maxwell C > > Sent: Monday, July 07, 2014 1:51 PM > > To: 'user@storm.incubator.apache.org' > > Subject: storm-kafka integration duplication of messages > > > > > > > > Working with a kafkaSpout feeding a storm topology, and dumping back out > of the topology to kafka and hdfs (duplicating sending to AllGrouping), I > am seeing a great deal of duplicate messages coming through the flow. > > > > > > > > The topology is fairly trivial in scope currently, as it is just > KafkaSpout -> Bolt to deserialize a java object and emit a field as string > -> ( HdfsBolt from the pgoetz project AND KafkaBolt to outgoing queue). > > > > The latter portion, the hdfsbolt/output to kafka doesn’t seem to be the > source of the problem, as we’re seeing duplicate messages incoming in the > deserialization, so they just suffer the symptoms. > > > > The deserialization bolt is too simple that I don’t think it could be > the problem… it gets a tuple, gets the byte array, deserializes, emits, > acks. > > > > > > > > So that leaves the KafkaSpout as missing some vital configuration to > prevent the massive duplication we are seeing of incoming messages. The > configuration is almost identical to the few examples I’ve seen around, in > the Michael Knoll tutorial, as well in the Nathan Marz git. > > > > > > > > //** > > > > ZkHosts hosts = new ZkHosts(brokerZkStr); > > > > > > > > SpoutConfig spoutConfig = new SpoutConfig(hosts, topic, zkRoot, > > > >spoutConfigID); > > > > spoutConfig.scheme = new RawMultiScheme(); > > > > > > > > KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig); > > > > topologyBuilder.setSpout(spoutID, kafkaSpout, spout_parallelism_hint); > > > > //* > > > > > > > > > > > > Can anyone explain what factors would cause the duplication of messages > that I’m seeing? Should I be looking more into kafka than into the > storm-kafka spout? > > > > > > > > Regards, > > > > Maxwell E. > > > > > > > > > > This message, and any attachments, is for the intended recipient(s) > only, may contain information that is privileged, confidential and/or > proprietary and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. >
Fwd: FW: storm-kafka integration duplication of messages
> From: Evers, Maxwell C > Sent: Monday, July 07, 2014 1:51 PM > To: 'user@storm.incubator.apache.org' > Subject: storm-kafka integration duplication of messages > > > > Working with a kafkaSpout feeding a storm topology, and dumping back out of the topology to kafka and hdfs (duplicating sending to AllGrouping), I am seeing a great deal of duplicate messages coming through the flow. > > > > The topology is fairly trivial in scope currently, as it is just KafkaSpout -> Bolt to deserialize a java object and emit a field as string -> ( HdfsBolt from the pgoetz project AND KafkaBolt to outgoing queue). > > The latter portion, the hdfsbolt/output to kafka doesn’t seem to be the source of the problem, as we’re seeing duplicate messages incoming in the deserialization, so they just suffer the symptoms. > > The deserialization bolt is too simple that I don’t think it could be the problem… it gets a tuple, gets the byte array, deserializes, emits, acks. > > > > So that leaves the KafkaSpout as missing some vital configuration to prevent the massive duplication we are seeing of incoming messages. The configuration is almost identical to the few examples I’ve seen around, in the Michael Knoll tutorial, as well in the Nathan Marz git. > > > > //** > > ZkHosts hosts = new ZkHosts(brokerZkStr); > > > > SpoutConfig spoutConfig = new SpoutConfig(hosts, topic, zkRoot, > >spoutConfigID); > > spoutConfig.scheme = new RawMultiScheme(); > > > > KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig); > > topologyBuilder.setSpout(spoutID, kafkaSpout, spout_parallelism_hint); > > //* > > > > > > Can anyone explain what factors would cause the duplication of messages that I’m seeing? Should I be looking more into kafka than into the storm-kafka spout? > > > > Regards, > > Maxwell E. > > > > > This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Storm-kafka Integration
Dear Komal, I used the storm-kafka version mentioned in the link. http://anisnasir.wordpress.com/2014/05/25/apache-storm-kafka-a-nice-choice/ Regards Anis On Mon, Jun 2, 2014 at 1:49 PM, Komal Thombare wrote: > Hi Andres, > > Still i am getting the same error. > > > Thanks and Regards, > > Komal Thombare > > > -Andres Gomez wrote: - > > To: user@storm.incubator.apache.org > From: Andres Gomez > Date: 06/02/2014 04:29PM > > Subject: Re: Storm-kafka Integration > > Hi again, > > I advise you to use the repo kafka happens to you: > > > https://github.com/apache/incubator-storm/tree/master/external/storm-kafka > > > 1. make a clone of this project > 2. mvm package — generate a jar of project > 3. mvn install — install the project on local repository > > Regards, > > Andres > > El 02/06/2014, a las 12:56, Komal Thombare > escribió: > > Hi, > > I have downloaded the storm-kafka jar from the below link: > http://repo1.maven.org/maven2/com/n3twork/storm/storm-kafka/20140521/ > > Now I have made changes in the storm word count topology and used > KafkaSpout. > While running topology in Local mode I get following error: > > java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy > at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) > ~[storm-kafka-20140521.jar:na] > at > backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116.invoke(executor.clj:519) > ~[na:na] > at backtype.storm.util$async_loop$fn__390.invoke(util.clj:431) ~[na:na] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na] > at java.lang.Thread.run(Thread.java:636) [na:1.6.0_17] > Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy > at java.net.URLClassLoader$1.run(URLClassLoader.java:217) > ~[na:1.6.0_17] > at java.security.AccessController.doPrivileged(Native Method) > ~[na:1.6.0_17] > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > ~[na:1.6.0_17] > at java.lang.ClassLoader.loadClass(ClassLoader.java:319) ~[na:1.6.0_17] > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) > ~[na:1.6.0_17] > at java.lang.ClassLoader.loadClass(ClassLoader.java:264) ~[na:1.6.0_17] > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:332) > ~[na:1.6.0_17] > > Though the curator-client-1.0.1.jar is present in the lib of storm and I > have included same in my buildpath. > > > Thanks and Regards, > > Komal Thombare > > > -Andres Gomez wrote: - > > To: user@storm.incubator.apache.org > From: Andres Gomez > Date: 06/02/2014 02:33PM > Subject: Re: Storm-kafka Integration > > You should make a clone of this project: > > https://github.com/apache/incubator-storm/tree/master/external/storm-kafka > > and do “mvn install”, I suposse you use kafka 0.8.+ > > and then you do this: > > SpoutConfig spoutConfig = new SpoutConfig(); > > spoutConfig.zkServers = “localhost”; //zookeeper Host > spoutConfig.zkPort = 2181; //zookeeper port > spoutConfig.zkRoot = “/kafkaStorm/“; // zookeeper path > spoutConfig.id = “storm”; // zookeeper id > > > // This config is a example!! > > KafkaSpout spout = new KafkaSpout(spoutConfig); > > > El 02/06/2014, a las 10:56, Joe Stein escribió: > > Please take a look at > http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/#state-of-the-integration-game > > There is a github project too > https://github.com/miguno/kafka-storm-starter > > This covers latest Storm, Kafka and Avro. > > /*** > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > / > > > On Mon, Jun 2, 2014 at 4:55 AM, Komal Thombare > wrote: > >> Hi Deepak, >> >> Yes i have. I have also got the storm-contrib source code, but then I am >> unaware of how to compile it. >> >> >> Thanks and Regards, >> >> Komal Thombare >> Tata Consultancy Services Limited >> Ph:- 086-55388772 >> Mail-to: komal.thomb...@tcs.com >> Website: http://www.tcs.com >> >> Experience certainty. IT Services >> Business Solutions >> Consulting >> >> >> -Deepak Sharma wrote: - >> >> To: user >> From: Deepak Sharma >> Date: 06/02/2014 02:22PM >> Subject: Re: S
Re: Storm-kafka Integration
Hi Andres, Still i am getting the same error. Thanks and Regards, Komal Thombare -Andres Gomez wrote: - To: user@storm.incubator.apache.org From: Andres Gomez Date: 06/02/2014 04:29PM Subject: Re: Storm-kafka Integration Hi again, I advise you to use the repo kafka happens to you: https://github.com/apache/incubator-storm/tree/master/external/storm-kafka 1. make a clone of this project 2. mvm package — generate a jar of project 3. mvn install — install the project on local repository Regards, Andres El 02/06/2014, a las 12:56, Komal Thombare escribió: Hi, I have downloaded the storm-kafka jar from the below link: http://repo1.maven.org/maven2/com/n3twork/storm/storm-kafka/20140521/ Now I have made changes in the storm word count topology and used KafkaSpout. While running topology in Local mode I get following error: java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-20140521.jar:na] at backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116.invoke(executor.clj:519) ~[na:na] at backtype.storm.util$async_loop$fn__390.invoke(util.clj:431) ~[na:na] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na] at java.lang.Thread.run(Thread.java:636) [na:1.6.0_17] Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy at java.net.URLClassLoader$1.run(URLClassLoader.java:217) ~[na:1.6.0_17] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.6.0_17] at java.net.URLClassLoader.findClass(URLClassLoader.java:205) ~[na:1.6.0_17] at java.lang.ClassLoader.loadClass(ClassLoader.java:319) ~[na:1.6.0_17] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) ~[na:1.6.0_17] at java.lang.ClassLoader.loadClass(ClassLoader.java:264) ~[na:1.6.0_17] at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:332) ~[na:1.6.0_17] Though the curator-client-1.0.1.jar is present in the lib of storm and I have included same in my buildpath. Thanks and Regards, Komal Thombare -Andres Gomez wrote: - To: user@storm.incubator.apache.org From: Andres Gomez Date: 06/02/2014 02:33PM Subject: Re: Storm-kafka Integration You should make a clone of this project: https://github.com/apache/incubator-storm/tree/master/external/storm-kafka and do “mvn install”, I suposse you use kafka 0.8.+ and then you do this: SpoutConfig spoutConfig = new SpoutConfig(); spoutConfig.zkServers = “localhost”; //zookeeper Host spoutConfig.zkPort = 2181; //zookeeper port spoutConfig.zkRoot = “/kafkaStorm/“; // zookeeper path spoutConfig.id = “storm”; // zookeeper id // This config is a example!! KafkaSpout spout = new KafkaSpout(spoutConfig); El 02/06/2014, a las 10:56, Joe Stein escribió: Please take a look at http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/#state-of-the-integration-game There is a github project too https://github.com/miguno/kafka-storm-starter This covers latest Storm, Kafka and Avro. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop / On Mon, Jun 2, 2014 at 4:55 AM, Komal Thombare wrote: Hi Deepak, Yes i have. I have also got the storm-contrib source code, but then I am unaware of how to compile it. Thanks and Regards, Komal Thombare Tata Consultancy Services Limited Ph:- 086-55388772 Mail-to: komal.thomb...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting -Deepak Sharma wrote: - To: user From: Deepak Sharma Date: 06/02/2014 02:22PM Subject: Re: Storm-kafka Integration Hi Komal Have you looked at KafkaSpout? Thanks Deepak On Mon, Jun 2, 2014 at 2:13 PM, Komal Thombare wrote: Hi, I am trying to integrate storm with kafka, but did not get any proper documentation to do so. Can anyone please help me for getting started with the integration. Thanks and Regards, Komal Thombare =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Thanks Deepak www.bigdatabig.com www.keosha.net
Re: Storm-kafka Integration
Hi again, I advise you to use the repo kafka happens to you: > https://github.com/apache/incubator-storm/tree/master/external/storm-kafka 1. make a clone of this project 2. mvm package — generate a jar of project 3. mvn install — install the project on local repository Regards, Andres El 02/06/2014, a las 12:56, Komal Thombare escribió: > Hi, > > I have downloaded the storm-kafka jar from the below link: > http://repo1.maven.org/maven2/com/n3twork/storm/storm-kafka/20140521/ > > Now I have made changes in the storm word count topology and used KafkaSpout. > While running topology in Local mode I get following error: > > java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy > at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) > ~[storm-kafka-20140521.jar:na] > at > backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116.invoke(executor.clj:519) > ~[na:na] > at backtype.storm.util$async_loop$fn__390.invoke(util.clj:431) ~[na:na] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na] > at java.lang.Thread.run(Thread.java:636) [na:1.6.0_17] > Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy > at java.net.URLClassLoader$1.run(URLClassLoader.java:217) ~[na:1.6.0_17] > at java.security.AccessController.doPrivileged(Native Method) > ~[na:1.6.0_17] > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > ~[na:1.6.0_17] > at java.lang.ClassLoader.loadClass(ClassLoader.java:319) ~[na:1.6.0_17] > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) > ~[na:1.6.0_17] > at java.lang.ClassLoader.loadClass(ClassLoader.java:264) ~[na:1.6.0_17] > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:332) > ~[na:1.6.0_17] > > Though the curator-client-1.0.1.jar is present in the lib of storm and I have > included same in my buildpath. > > > Thanks and Regards, > > Komal Thombare > > > -Andres Gomez wrote: - > To: user@storm.incubator.apache.org > From: Andres Gomez > Date: 06/02/2014 02:33PM > Subject: Re: Storm-kafka Integration > > You should make a clone of this project: > > https://github.com/apache/incubator-storm/tree/master/external/storm-kafka > > and do “mvn install”, I suposse you use kafka 0.8.+ > > and then you do this: > > SpoutConfig spoutConfig = new SpoutConfig(); > > spoutConfig.zkServers = “localhost”; //zookeeper Host > spoutConfig.zkPort = 2181; //zookeeper port > spoutConfig.zkRoot = “/kafkaStorm/“; // zookeeper path > spoutConfig.id = “storm”; // zookeeper id > > // This config is a example!! > > KafkaSpout spout = new KafkaSpout(spoutConfig); > > > El 02/06/2014, a las 10:56, Joe Stein escribió: > >> Please take a look at >> http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/#state-of-the-integration-game >> >> There is a github project too https://github.com/miguno/kafka-storm-starter >> >> This covers latest Storm, Kafka and Avro. >> >> /*** >> Joe Stein >> Founder, Principal Consultant >> Big Data Open Source Security LLC >> http://www.stealth.ly >> Twitter: @allthingshadoop >> / >> >> >> On Mon, Jun 2, 2014 at 4:55 AM, Komal Thombare >> wrote: >> Hi Deepak, >> >> Yes i have. I have also got the storm-contrib source code, but then I am >> unaware of how to compile it. >> >> >> Thanks and Regards, >> >> Komal Thombare >> Tata Consultancy Services Limited >> Ph:- 086-55388772 >> Mail-to: komal.thomb...@tcs.com >> Website: http://www.tcs.com >> >> Experience certainty. IT Services >> Business Solutions >> Consulting >> >> >> -Deepak Sharma wrote: - >> To: user >> From: Deepak Sharma >> Date: 06/02/2014 02:22PM >> Subject: Re: Storm-kafka Integration >> >> >> Hi Komal >> Have you looked at KafkaSpout? >> >> Thanks >> Deepak >> >> >> On Mon, Jun 2, 2014 at 2:13 PM, Komal Thombare >> wrote: >> Hi, >> >> I am trying to integrate storm with kafka, but did not get any proper >> documentation to do so. >> Can anyone please help me for getting started with the integration. >> >> Thanks and Regards, >> >> Komal Thombare >> >> =-=-= >> Notice: Th
Re: Storm-kafka Integration
Hi, I have downloaded the storm-kafka jar from the below link: http://repo1.maven.org/maven2/com/n3twork/storm/storm-kafka/20140521/ Now I have made changes in the storm word count topology and used KafkaSpout. While running topology in Local mode I get following error: java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-20140521.jar:na] at backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116.invoke(executor.clj:519) ~[na:na] at backtype.storm.util$async_loop$fn__390.invoke(util.clj:431) ~[na:na] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na] at java.lang.Thread.run(Thread.java:636) [na:1.6.0_17] Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy at java.net.URLClassLoader$1.run(URLClassLoader.java:217) ~[na:1.6.0_17] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.6.0_17] at java.net.URLClassLoader.findClass(URLClassLoader.java:205) ~[na:1.6.0_17] at java.lang.ClassLoader.loadClass(ClassLoader.java:319) ~[na:1.6.0_17] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) ~[na:1.6.0_17] at java.lang.ClassLoader.loadClass(ClassLoader.java:264) ~[na:1.6.0_17] at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:332) ~[na:1.6.0_17] Though the curator-client-1.0.1.jar is present in the lib of storm and I have included same in my buildpath. Thanks and Regards, Komal Thombare -Andres Gomez wrote: - To: user@storm.incubator.apache.org From: Andres Gomez Date: 06/02/2014 02:33PM Subject: Re: Storm-kafka Integration You should make a clone of this project: https://github.com/apache/incubator-storm/tree/master/external/storm-kafka and do “mvn install”, I suposse you use kafka 0.8.+ and then you do this: SpoutConfig spoutConfig = new SpoutConfig(); spoutConfig.zkServers = “localhost”; //zookeeper Host spoutConfig.zkPort = 2181; //zookeeper port spoutConfig.zkRoot = “/kafkaStorm/“; // zookeeper path spoutConfig.id = “storm”; // zookeeper id // This config is a example!! KafkaSpout spout = new KafkaSpout(spoutConfig); El 02/06/2014, a las 10:56, Joe Stein escribió: Please take a look at http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/#state-of-the-integration-game There is a github project too https://github.com/miguno/kafka-storm-starter This covers latest Storm, Kafka and Avro. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop / On Mon, Jun 2, 2014 at 4:55 AM, Komal Thombare wrote: Hi Deepak, Yes i have. I have also got the storm-contrib source code, but then I am unaware of how to compile it. Thanks and Regards, Komal Thombare Tata Consultancy Services Limited Ph:- 086-55388772 Mail-to: komal.thomb...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting -Deepak Sharma wrote: - To: user From: Deepak Sharma Date: 06/02/2014 02:22PM Subject: Re: Storm-kafka Integration Hi Komal Have you looked at KafkaSpout? Thanks Deepak On Mon, Jun 2, 2014 at 2:13 PM, Komal Thombare wrote: Hi, I am trying to integrate storm with kafka, but did not get any proper documentation to do so. Can anyone please help me for getting started with the integration. Thanks and Regards, Komal Thombare =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Thanks Deepak www.bigdatabig.com www.keosha.net
Re: Storm-kafka Integration
You should make a clone of this project: https://github.com/apache/incubator-storm/tree/master/external/storm-kafka and do “mvn install”, I suposse you use kafka 0.8.+ and then you do this: SpoutConfig spoutConfig = new SpoutConfig(); spoutConfig.zkServers = “localhost”; //zookeeper Host spoutConfig.zkPort = 2181; //zookeeper port spoutConfig.zkRoot = “/kafkaStorm/“; // zookeeper path spoutConfig.id = “storm”; // zookeeper id // This config is a example!! KafkaSpout spout = new KafkaSpout(spoutConfig); El 02/06/2014, a las 10:56, Joe Stein escribió: > Please take a look at > http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/#state-of-the-integration-game > > There is a github project too https://github.com/miguno/kafka-storm-starter > > This covers latest Storm, Kafka and Avro. > > /*** > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop > / > > > On Mon, Jun 2, 2014 at 4:55 AM, Komal Thombare wrote: > Hi Deepak, > > Yes i have. I have also got the storm-contrib source code, but then I am > unaware of how to compile it. > > > Thanks and Regards, > > Komal Thombare > Tata Consultancy Services Limited > Ph:- 086-55388772 > Mail-to: komal.thomb...@tcs.com > Website: http://www.tcs.com > > Experience certainty. IT Services > Business Solutions > Consulting > > > -----Deepak Sharma wrote: - > To: user > From: Deepak Sharma > Date: 06/02/2014 02:22PM > Subject: Re: Storm-kafka Integration > > > Hi Komal > Have you looked at KafkaSpout? > > Thanks > Deepak > > > On Mon, Jun 2, 2014 at 2:13 PM, Komal Thombare wrote: > Hi, > > I am trying to integrate storm with kafka, but did not get any proper > documentation to do so. > Can anyone please help me for getting started with the integration. > > Thanks and Regards, > > Komal Thombare > > =-=-= > Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > > > > > > -- > Thanks > Deepak > www.bigdatabig.com > www.keosha.net >
Re: Storm-kafka Integration
Please take a look at http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/#state-of-the-integration-game There is a github project too https://github.com/miguno/kafka-storm-starter This covers latest Storm, Kafka and Avro. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Mon, Jun 2, 2014 at 4:55 AM, Komal Thombare wrote: > Hi Deepak, > > Yes i have. I have also got the storm-contrib source code, but then I am > unaware of how to compile it. > > > Thanks and Regards, > > Komal Thombare > Tata Consultancy Services Limited > Ph:- 086-55388772 > Mail-to: komal.thomb...@tcs.com > Website: http://www.tcs.com > > Experience certainty. IT Services > Business Solutions > Consulting > > > -Deepak Sharma wrote: - > > To: user > From: Deepak Sharma > Date: 06/02/2014 02:22PM > Subject: Re: Storm-kafka Integration > > > Hi Komal > Have you looked at KafkaSpout? > > Thanks > Deepak > > > On Mon, Jun 2, 2014 at 2:13 PM, Komal Thombare > wrote: > >> Hi, >> >> I am trying to integrate storm with kafka, but did not get any proper >> documentation to do so. >> Can anyone please help me for getting started with the integration. >> >> Thanks and Regards, >> >> Komal Thombare >> >> =-=-= >> Notice: The information contained in this e-mail >> message and/or attachments to it may contain >> confidential or privileged information. If you are >> not the intended recipient, any dissemination, use, >> review, distribution, printing or copying of the >> information contained in this e-mail message >> and/or attachments to it are strictly prohibited. If >> you have received this communication in error, >> please notify us by reply e-mail or telephone and >> immediately and permanently delete the message >> and any attachments. Thank you >> >> > > > -- > Thanks > Deepak > www.bigdatabig.com > www.keosha.net > >
Re: Storm-kafka Integration
Hi Deepak, Yes i have. I have also got the storm-contrib source code, but then I am unaware of how to compile it. Thanks and Regards, Komal Thombare Tata Consultancy Services Limited Ph:- 086-55388772 Mail-to: komal.thomb...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting -Deepak Sharma wrote: - To: user From: Deepak Sharma Date: 06/02/2014 02:22PM Subject: Re: Storm-kafka Integration Hi Komal Have you looked at KafkaSpout? Thanks Deepak On Mon, Jun 2, 2014 at 2:13 PM, Komal Thombare wrote: Hi, I am trying to integrate storm with kafka, but did not get any proper documentation to do so. Can anyone please help me for getting started with the integration. Thanks and Regards, Komal Thombare =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Thanks Deepak www.bigdatabig.com www.keosha.net
Re: Storm-kafka Integration
Hi Komal Have you looked at KafkaSpout? Thanks Deepak On Mon, Jun 2, 2014 at 2:13 PM, Komal Thombare wrote: > Hi, > > I am trying to integrate storm with kafka, but did not get any proper > documentation to do so. > Can anyone please help me for getting started with the integration. > > Thanks and Regards, > > Komal Thombare > > =-=-= > Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > > -- Thanks Deepak www.bigdatabig.com www.keosha.net
Re: Storm-kafka Integration
Only Storm Thanks and Regards, Komal Thombare Tata Consultancy Services Limited Ph:- 086-55388772 Mail-to: komal.thomb...@tcs.com Website: http://www.tcs.com Experience certainty. IT Services Business Solutions Consulting -Andres Gomez wrote: - To: user@storm.incubator.apache.org From: Andres Gomez Date: 06/02/2014 02:17PM Subject: Re: Storm-kafka Integration Do you use trindent or only storm??? Regards, Andres El 02/06/2014, a las 10:43, Komal Thombare escribió: Hi, I am trying to integrate storm with kafka, but did not get any proper documentation to do so. Can anyone please help me for getting started with the integration. Thanks and Regards, Komal Thombare =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
Re: Storm-kafka Integration
Do you use trindent or only storm??? Regards, Andres El 02/06/2014, a las 10:43, Komal Thombare escribió: > Hi, > > I am trying to integrate storm with kafka, but did not get any proper > documentation to do so. > Can anyone please help me for getting started with the integration. > > Thanks and Regards, > > Komal Thombare > > =-=-= > Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > >
Storm-kafka Integration
Hi, I am trying to integrate storm with kafka, but did not get any proper documentation to do so. Can anyone please help me for getting started with the integration. Thanks and Regards, Komal Thombare =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
