Robert,
I got it working for 1.0.0.
balaji
On Mon, Mar 14, 2016 at 4:41 PM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com> wrote:
> Yep the same issue as before(class not found) with flink 0.10.2 with
> scala version 2.11. I was not able to use scala 2.10 since connector for
> flink_connector_kafka for 0.10.2 is not available.
>
> balaji
>
> On Mon, Mar 14, 2016 at 4:20 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> Yes figured that out, thanks for point that, my bad. I have put back
>> 0.10.2 as flink version, will try to reproduce the problem again, this time
>> I have explicitly called out the scala version as 2.11.
>>
>>
>> On Mon, Mar 14, 2016 at 4:14 PM, Robert Metzger <rmetz...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> flink-connector-kafka_ doesn't exist for 1.0.0. You have to use either
>>> flink-connector-kafka-0.8_ or flink-connector-kafka-0.9_
>>>
>>>
>>> On Mon, Mar 14, 2016 at 11:17 AM, Balaji Rajagopalan <
>>> balaji.rajagopa...@olacabs.com> wrote:
>>>
>>>> What I noticied was that, if I remove the dependency on
>>>> flink-connector-kafka so it is clearly to do something with that
>>>> dependency.
>>>>
>>>>
>>>> On Mon, Mar 14, 2016 at 3:46 PM, Balaji Rajagopalan <
>>>> balaji.rajagopa...@olacabs.com> wrote:
>>>>
>>>>> Robert,
>>>>> I have moved on to latest version of flink of 1.0.0 hoping that
>>>>> will solve my problem with kafka connector . Here is my pom.xml but now I
>>>>> cannot get the code compiled.
>>>>>
>>>>> [ERROR] Failed to execute goal
>>>>> net.alchim31.maven:scala-maven-plugin:3.2.1:compile (scala-compile-first)
>>>>> on project flink-streaming-demo: Execution scala-compile-first of goal
>>>>> net.alchim31.maven:scala-maven-plugin:3.2.1:compile failed: For artifact
>>>>> {null:null:null:jar}: The groupId cannot be empty. -> [Help 1]
>>>>>
>>>>> I read about the above errors in most cases people where able to
>>>>> overcome is by deleting the .m2 directory, and that did not fix the issue
>>>>> for me.
>>>>>
>>>>> What I noticied was that, if I remove the dependency on
>>>>>
>>>>> Here is my pom.xml
>>>>>
>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>> <!--
>>>>> Copyright 2015 data Artisans GmbH
>>>>>
>>>>> Licensed under the Apache License, Version 2.0 (the "License");
>>>>> you may not use this file except in compliance with the License.
>>>>> You may obtain a copy of the License at
>>>>>
>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>
>>>>> Unless required by applicable law or agreed to in writing, software
>>>>> distributed under the License is distributed on an "AS IS" BASIS,
>>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>>>> See the License for the specific language governing permissions and
>>>>> limitations under the License.
>>>>> -->
>>>>> <project xmlns="http://maven.apache.org/POM/4.0.0";
>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>>>>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>>>>> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>>>>> <modelVersion>4.0.0</modelVersion>
>>>>>
>>>>> <groupId>com.dataArtisans</groupId>
>>>>> <artifactId>flink-streaming-demo</artifactId>
>>>>> <version>0.1</version>
>>>>> <packaging>jar</packaging>
>>>>>
>>>>> <name>Flink Streaming Demo</name>
>>>>> <url>http://www.data-artisans.com</url>
>>>>>
>>>>> <properties>
>>>>> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
>>>>> <slf4j.version>1.7.12</slf4j.version>
>>>>> <flink.version>1.0.0</flink.version>
>>>>> <scala.version>2.10</scala.version>
>>>>> </properties>
>>>>>
>>>>> <dependencies>
>>>>>
>>>>>
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.apache.flink</groupId>
>>>>> <artifactId>flink-streaming-scala_${scala.version}</artifactId>
>>>>> <version>${flink.version}</version>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.apache.flink</groupId>
>>>>> <artifactId>flink-runtime-web_${scala.version}</artifactId>
>>>>> <version>${flink.version}</version>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.elasticsearch</groupId>
>>>>> <artifactId>elasticsearch</artifactId>
>>>>> <version>1.7.3</version>
>>>>> <scope>compile</scope>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>> <groupId>joda-time</groupId>
>>>>> <artifactId>joda-time</artifactId>
>>>>> <version>2.7</version>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.apache.kafka</groupId>
>>>>> <artifactId>kafka_${scala.version}</artifactId>
>>>>> <version>0.8.2.0</version>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.apache.flink</groupId>
>>>>> <artifactId>flink-connector-kafka_${scala.version}</artifactId>
>>>>> <version>${flink.version}</version>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.json4s</groupId>
>>>>> <artifactId>json4s-native_${scala.version}</artifactId>
>>>>> <version>3.3.0</version>
>>>>> </dependency>
>>>>>
>>>>>
>>>>> </dependencies>
>>>>>
>>>>> <build>
>>>>> <plugins>
>>>>>
>>>>> <!-- Scala Compiler -->
>>>>> <plugin>
>>>>> <groupId>net.alchim31.maven</groupId>
>>>>> <artifactId>scala-maven-plugin</artifactId>
>>>>> <version>3.2.1</version>
>>>>> <executions>
>>>>> <!-- Run scala compiler in the process-resources phase, so
>>>>> that dependencies on
>>>>> scala classes can be resolved later in the (Java)
>>>>> compile phase -->
>>>>> <execution>
>>>>> <id>scala-compile-first</id>
>>>>> <phase>process-resources</phase>
>>>>> <goals>
>>>>> <goal>compile</goal>
>>>>> </goals>
>>>>> </execution>
>>>>>
>>>>> <!-- Run scala compiler in the process-test-resources
>>>>> phase, so that dependencies on
>>>>> scala classes can be resolved later in the (Java)
>>>>> test-compile phase -->
>>>>> <execution>
>>>>> <id>scala-test-compile</id>
>>>>> <phase>process-test-resources</phase>
>>>>> <goals>
>>>>> <goal>testCompile</goal>
>>>>> </goals>
>>>>> </execution>
>>>>> </executions>
>>>>> <configuration>
>>>>> <jvmArgs>
>>>>> <jvmArg>-Xms128m</jvmArg>
>>>>> <jvmArg>-Xmx512m</jvmArg>
>>>>> </jvmArgs>
>>>>> </configuration>
>>>>> </plugin>
>>>>>
>>>>> <plugin>
>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>> <artifactId>maven-dependency-plugin</artifactId>
>>>>> <version>2.9</version>
>>>>> <executions>
>>>>> <execution>
>>>>> <id>unpack</id>
>>>>> <!-- executed just before the package phase -->
>>>>> <phase>prepare-package</phase>
>>>>> <goals>
>>>>> <goal>unpack</goal>
>>>>> </goals>
>>>>> <configuration>
>>>>> <artifactItems>
>>>>> <!-- For Flink connector classes -->
>>>>> <artifactItem>
>>>>> <groupId>org.apache.flink</groupId>
>>>>>
>>>>> <artifactId>flink-connector-kafka_${scala.version}</artifactId>
>>>>> <version>1.0.0</version>
>>>>> <type>jar</type>
>>>>> <overWrite>false</overWrite>
>>>>>
>>>>> <outputDirectory>${project.build.directory}/classes</outputDirectory>
>>>>> <includes>org/apache/flink/**</includes>
>>>>> </artifactItem>
>>>>> <!-- For Kafka API classes -->
>>>>> <artifactItem>
>>>>> <groupId>org.apache.kafka</groupId>
>>>>> <artifactId>kafka_${scala.version}</artifactId>
>>>>> <version>0.8.2.0</version>
>>>>> <type>jar</type>
>>>>> <overWrite>false</overWrite>
>>>>>
>>>>> <outputDirectory>${project.build.directory}/classes</outputDirectory>
>>>>> <includes>kafka/**</includes>
>>>>> </artifactItem>
>>>>> </artifactItems>
>>>>> </configuration>
>>>>> </execution>
>>>>> </executions>
>>>>> </plugin>
>>>>>
>>>>> <!--plugin>
>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>> <artifactId>maven-jar-plugin</artifactId>
>>>>> <executions>
>>>>>
>>>>> <execution>
>>>>> <id>MBoxParser</id>
>>>>> <phase>package</phase>
>>>>> <goals>
>>>>> <goal>jar</goal>
>>>>> </goals>
>>>>>
>>>>> <configuration>
>>>>> <classifier>MBoxParser</classifier>
>>>>>
>>>>> <archive>
>>>>> <manifestEntries>
>>>>>
>>>>> <main-class>com.dataartisans.flinkTraining.dataSetPreparation.MBoxParser</main-class>
>>>>> </manifestEntries>
>>>>> </archive>
>>>>>
>>>>> <includes>
>>>>> <include>**/MBoxParser.class</include>
>>>>> <include>**/MBoxParser$*.class</include>
>>>>> </includes>
>>>>> </configuration>
>>>>> </execution>
>>>>>
>>>>> </executions>
>>>>> </plugin-->
>>>>>
>>>>> <plugin>
>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>> <artifactId>maven-compiler-plugin</artifactId>
>>>>> <version>3.1</version>
>>>>> <configuration>
>>>>> <source>1.8</source> <!-- If you want to use Java 8,
>>>>> change this to "1.8" -->
>>>>> <target>1.8</target> <!-- If you want to use Java 8,
>>>>> change this to "1.8" -->
>>>>> </configuration>
>>>>> </plugin>
>>>>>
>>>>> <plugin>
>>>>> <groupId>org.apache.rat</groupId>
>>>>> <artifactId>apache-rat-plugin</artifactId>
>>>>> <version>0.10</version><!--$NO-MVN-MAN-VER$-->
>>>>> <inherited>false</inherited>
>>>>> <executions>
>>>>> <execution>
>>>>> <phase>verify</phase>
>>>>> <goals>
>>>>> <goal>check</goal>
>>>>> </goals>
>>>>> </execution>
>>>>> </executions>
>>>>> <configuration>
>>>>> <excludeSubProjects>false</excludeSubProjects>
>>>>> <numUnapprovedLicenses>0</numUnapprovedLicenses>
>>>>> <licenses>
>>>>> <!-- Enforce this license:
>>>>> Copyright 2015 data Artisans GmbH
>>>>>
>>>>> Licensed under the Apache License, Version 2.0 (the
>>>>> "License");
>>>>> you may not use this file except in compliance with
>>>>> the License.
>>>>> You may obtain a copy of the License at
>>>>>
>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>
>>>>> Unless required by applicable law or agreed to in
>>>>> writing, software
>>>>> distributed under the License is distributed on an
>>>>> "AS IS" BASIS,
>>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
>>>>> express or implied.
>>>>> See the License for the specific language governing
>>>>> permissions and
>>>>> limitations under the License.
>>>>> -->
>>>>> <license
>>>>> implementation="org.apache.rat.analysis.license.SimplePatternBasedLicense">
>>>>> <licenseFamilyCategory>AL2 </licenseFamilyCategory>
>>>>> <licenseFamilyName>Apache License
>>>>> 2.0</licenseFamilyName>
>>>>> <notes />
>>>>> <patterns>
>>>>> <pattern>Copyright 2015 data Artisans
>>>>> GmbH</pattern>
>>>>> <pattern>Licensed under the Apache License,
>>>>> Version 2.0 (the "License");</pattern>
>>>>> </patterns>
>>>>> </license>
>>>>> </licenses>
>>>>> <licenseFamilies>
>>>>> <licenseFamily
>>>>> implementation="org.apache.rat.license.SimpleLicenseFamily">
>>>>> <familyName>Apache License 2.0</familyName>
>>>>> </licenseFamily>
>>>>> </licenseFamilies>
>>>>> <excludes>
>>>>> <!-- Additional files like .gitignore etc.-->
>>>>> <exclude>**/.*</exclude>
>>>>> <exclude>**/*.prefs</exclude>
>>>>> <exclude>**/*.properties</exclude>
>>>>> <exclude>**/*.log</exclude>
>>>>> <exclude>*.txt/**</exclude>
>>>>> <!-- Administrative files in the main trunk. -->
>>>>> <exclude>**/README.md</exclude>
>>>>> <exclude>CHANGELOG</exclude>
>>>>> <!-- Build files -->
>>>>> <exclude>**/*.iml</exclude>
>>>>> <!-- Generated content -->
>>>>> <exclude>**/target/**</exclude>
>>>>> <exclude>**/build/**</exclude>
>>>>> </excludes>
>>>>> </configuration>
>>>>> </plugin>
>>>>>
>>>>> <plugin>
>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>> <artifactId>maven-checkstyle-plugin</artifactId>
>>>>> <version>2.12.1</version>
>>>>> <executions>
>>>>> <execution>
>>>>> <id>validate</id>
>>>>> <phase>validate</phase>
>>>>> <goals>
>>>>> <goal>check</goal>
>>>>> </goals>
>>>>> </execution>
>>>>> </executions>
>>>>> <configuration>
>>>>>
>>>>> <configLocation>/tools/maven/checkstyle.xml</configLocation>
>>>>> <logViolationsToConsole>true</logViolationsToConsole>
>>>>> </configuration>
>>>>> </plugin>
>>>>>
>>>>> </plugins>
>>>>>
>>>>> </build>
>>>>> </project>
>>>>>
>>>>>
>>>>> On Mon, Mar 14, 2016 at 3:15 PM, Robert Metzger <rmetz...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Can you send me the full build file to further investigate the issue?
>>>>>>
>>>>>> On Fri, Mar 11, 2016 at 4:56 PM, Balaji Rajagopalan <
>>>>>> balaji.rajagopa...@olacabs.com> wrote:
>>>>>>
>>>>>>> Robert,
>>>>>>> That did not fix it ( using flink and connector same version) .
>>>>>>> Tried with scala version 2.11, so will try to see scala 2.10 makes any
>>>>>>> difference.
>>>>>>>
>>>>>>> balaji
>>>>>>>
>>>>>>> On Fri, Mar 11, 2016 at 8:06 PM, Robert Metzger <rmetz...@apache.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> you have to use the same version for all dependencies from the
>>>>>>>> "org.apache.flink" group.
>>>>>>>>
>>>>>>>> You said these are the versions you are using:
>>>>>>>>
>>>>>>>> flink.version = 0.10.2
>>>>>>>> kafka.verison = 0.8.2
>>>>>>>> flink.kafka.connection.verion=0.9.1
>>>>>>>>
>>>>>>>> For the connector, you also need to use 0.10.2.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 11, 2016 at 9:56 AM, Balaji Rajagopalan <
>>>>>>>> balaji.rajagopa...@olacabs.com> wrote:
>>>>>>>>
>>>>>>>>> I am tyring to use the flink kafka connector, for this I have
>>>>>>>>> specified the kafka connector dependency and created a fat jar since
>>>>>>>>> default flink installation does not contain kafka connector jars. I
>>>>>>>>> have
>>>>>>>>> made sure that flink-streaming-demo-0.1.jar has the
>>>>>>>>> kafka.javaapi.consumer.SimpleConsumer.class but still I see the class
>>>>>>>>> not
>>>>>>>>> found exception.
>>>>>>>>>
>>>>>>>>> The code for kafka connector in flink.
>>>>>>>>>
>>>>>>>>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>>>>>>>> val prop:Properties = new Properties()
>>>>>>>>> prop.setProperty("zookeeper.connect","somezookeer:2181")
>>>>>>>>> prop.setProperty("group.id","some-group")
>>>>>>>>> prop.setProperty("bootstrap.servers","somebroker:9092")
>>>>>>>>>
>>>>>>>>> val stream = env
>>>>>>>>> .addSource(new FlinkKafkaConsumer082[String]("location", new
>>>>>>>>> SimpleStringSchema, prop))
>>>>>>>>>
>>>>>>>>> jar tvf flink-streaming-demo-0.1.jar | grep
>>>>>>>>> kafka.javaapi.consumer.SimpleConsumer
>>>>>>>>>
>>>>>>>>> 5111 Fri Mar 11 14:18:36 UTC 2016
>>>>>>>>> *kafka/javaapi/consumer/SimpleConsumer*.class
>>>>>>>>>
>>>>>>>>> flink.version = 0.10.2
>>>>>>>>> kafka.verison = 0.8.2
>>>>>>>>> flink.kafka.connection.verion=0.9.1
>>>>>>>>>
>>>>>>>>> The command that I use to run the flink program in yarn cluster is
>>>>>>>>> below,
>>>>>>>>>
>>>>>>>>> HADOOP_CONF_DIR=/etc/hadoop/conf /usr/share/flink/bin/flink run -c
>>>>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer -m
>>>>>>>>> yarn-cluster
>>>>>>>>> -yn 2 /home/balajirajagopalan/flink-streaming-demo-0.1.jar
>>>>>>>>>
>>>>>>>>> java.lang.NoClassDefFoundError:
>>>>>>>>> kafka/javaapi/consumer/SimpleConsumer
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.getPartitionsForTopic(FlinkKafkaConsumer.java:691)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:281)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer082.<init>(FlinkKafkaConsumer082.java:49)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer$.main(DriverEventConsumer.scala:53)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> com.dataartisans.flink_demo.examples.DriverEventConsumer.main(DriverEventConsumer.scala)
>>>>>>>>>
>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>
>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.program.Client.runBlocking(Client.java:252)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:676)
>>>>>>>>>
>>>>>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:326)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:978)
>>>>>>>>>
>>>>>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1028)
>>>>>>>>>
>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>> kafka.javaapi.consumer.SimpleConsumer
>>>>>>>>>
>>>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>>>>>
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>
>>>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>>>>>>>>>
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>
>>>>>>>>> ... 16 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any help appreciated.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> balaji
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
