As for locally I meant the machine that you use for development to see whether this works without parallelism. :-) No need to install stuff on your Namenode of course. Installing Kafka on a machine and having the Kafka Java dependencies available for Flink are two very different things. Try adding the following [1] to your maven pom. Then execute 'mvn assembly:assembly', this will produce a fat jar suffiexed jar-with-dependencies.jar. You should be able to run the example form that.
[1] https://github.com/mbalassi/flink-dataflow/blob/master/pom.xml#L286-296 On Thu, Jun 11, 2015 at 10:32 AM, Hawin Jiang <hawin.ji...@gmail.com> wrote: > Dear Marton > > What do you meaning for locally Eclipse with 'Run'. > Do you want to me to run it on Namenode? > But my namenode didn't install Kafka. I only installed Kafka on my data > node servers. > Do I need to install or copy Kafka jar on Namenode? Actually, I don't want > to install everything on Name node server. > Please advise me. > Thanks. > > > My Flink and Hadoop cluster info as below. > > Flink on NameNode > Kafka,Zookeeper and FLink slave1 on Datanode1 > Kafka,Zookeeper ,and Flink slave2 on Datanode2 > Kafka, Zookeeper and Flink slave3 on Datanode3 > > > > On Thu, Jun 11, 2015 at 1:16 AM, Márton Balassi <balassi.mar...@gmail.com> > wrote: > >> Dear Hawin, >> >> No problem, I am gald that you are giving our Kafka connector a try. :) >> The dependencies listed look good. Can you run the example locally from >> Eclipse with 'Run'? I suspect that maybe your Flink cluster does not have >> the access to the kafka dependency then. >> >> As a quick test you could copy the kafka jars to the lib folder of your >> Flink distribution on all the machines in your cluster. Everything that is >> there goes to the classpath of Flink. Another workaround with be to build a >> fat jar for your project containing all the dependencies with 'mvn >> assembly:assembly'. Neither of these are beautiful but would help tracking >> down the root cause. >> >> On Thu, Jun 11, 2015 at 10:04 AM, Hawin Jiang <hawin.ji...@gmail.com> >> wrote: >> >>> Dear Marton >>> >>> >>> >>> Thanks for supporting again. >>> >>> I am running these examples at the same project and I am using Eclipse >>> IDE to submit it to my Flink cluster. >>> >>> >>> >>> >>> >>> Here is my dependencies >>> >>> >>> ****************************************************************************** >>> >>> <dependencies> >>> >>> <dependency> >>> >>> <groupId>*junit*</groupId> >>> >>> <artifactId>*junit*</artifactId> >>> >>> <version>4.12</version> >>> >>> <scope>test</scope> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.flink</groupId> >>> >>> <artifactId>*flink*-java</artifactId> >>> >>> <version>0.9.0-milestone-1</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.flink</groupId> >>> >>> <artifactId>*flink*-clients</artifactId> >>> >>> <version>0.9.0-milestone-1</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.flink</groupId> >>> >>> <artifactId>*flink*-streaming-connectors</artifactId> >>> >>> <version>0.9.0-milestone-1</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.flink</groupId> >>> >>> <artifactId>*flink*-streaming-core</artifactId> >>> >>> <version>0.9.0-milestone-1</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.kafka</groupId> >>> >>> <artifactId>kafka_2.10</artifactId> >>> >>> <version>0.8.2.1</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.kafka</groupId> >>> >>> <artifactId>*kafka*-clients</artifactId> >>> >>> <version>0.8.2.1</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.hadoop</groupId> >>> >>> <artifactId>*hadoop*-*hdfs*</artifactId> >>> >>> <version>2.6.0</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.hadoop</groupId> >>> >>> <artifactId>*hadoop*-*auth*</artifactId> >>> >>> <version>2.6.0</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.hadoop</groupId> >>> >>> <artifactId>*hadoop*-common</artifactId> >>> >>> <version>2.6.0</version> >>> >>> </dependency> >>> >>> <dependency> >>> >>> <groupId>org.apache.hadoop</groupId> >>> >>> <artifactId>*hadoop*-core</artifactId> >>> >>> <version>1.2.1</version> >>> >>> </dependency> >>> >>> </dependencies> >>> >>> >>> >>> >>> ***************************************************************************************** >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Best regards >>> >>> Email: hawin.ji...@gmail.com >>> >>> >>> >>> *From:* Márton Balassi [mailto:balassi.mar...@gmail.com] >>> *Sent:* Thursday, June 11, 2015 12:58 AM >>> *To:* user@flink.apache.org >>> *Subject:* Re: Kafka0.8.2.1 + Flink0.9.0 issue >>> >>> >>> >>> Dear Hawin, >>> >>> >>> >>> This looks like a dependency issue, the java compiler does not find the >>> kafka dependency. How are you trying to run this example? Is it from an IDE >>> or submitting it to a flink cluster with bin/flink run? How do you define >>> your dependencies, do you use maven or sbt for instance? >>> >>> >>> >>> Best, >>> >>> >>> >>> Marton >>> >>> >>> >>> On Thu, Jun 11, 2015 at 9:43 AM, Hawin Jiang <hawin.ji...@gmail.com> >>> wrote: >>> >>> >>> >> >> >
