Thanks for your response. I have attached pom file. From: Chaitanya Chebolu [mailto:[email protected]] Sent: Tuesday, December 20, 2016 9:46 PM To: [email protected] Subject: Re: Property Name for Kafka config
John, This looks like a dependency issue. Could you please provide the details of which version of Kafka and Malhar library are you using? Regards, Chaitanya On Wed, Dec 21, 2016 at 2:39 AM, JOHN, BIBIN <[email protected]<mailto:[email protected]>> wrote: Thanks Chitanya. I was able to start job, but failing with below exception. 2016-12-20 12:44:47,768 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=hltd435.hydc.sbc.com:2181<http://hltd435.hydc.sbc.com:2181> sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@40dd3977<mailto:watcher=org.I0Itec.zkclient.ZkClient@40dd3977> 2016-12-20 12:44:47,782 INFO zkclient.ZkClient (ZkClient.java:waitForKeeperState(934)) - Waiting for keeper state SyncConnected 2016-12-20 12:44:47,786 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server localhost:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-20 12:44:47,787 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to localhost:2181, initiating session 2016-12-20 12:44:47,793 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1279)) - Session establishment complete on server localhost:2181, sessionid = 0x158fcc42050001a, negotiated timeout = 30000 2016-12-20 12:44:47,795 INFO zkclient.ZkClient (ZkClient.java:processStateChanged(711)) - zookeeper state changed (SyncConnected) 2016-12-20 12:44:47,797 ERROR stram.StreamingAppMaster (StreamingAppMaster.java:main(106)) - Exiting Application Master java.lang.NoSuchMethodError: kafka.utils.ZkUtils.getAllBrokersInCluster(Lorg/I0Itec/zkclient/ZkClient;)Lscala/collection/Seq; at com.datatorrent.contrib.kafka.KafkaMetadataUtil.getBrokers(KafkaMetadataUtil.java:113) at com.datatorrent.contrib.kafka.KafkaConsumer.initBrokers(KafkaConsumer.java:131) at com.datatorrent.contrib.kafka.AbstractKafkaInputOperator.definePartitions(AbstractKafkaInputOperator.java:488) at com.datatorrent.stram.plan.physical.PhysicalPlan.initPartitioning(PhysicalPlan.java:752) at com.datatorrent.stram.plan.physical.PhysicalPlan.addLogicalOperator(PhysicalPlan.java:1676) at com.datatorrent.stram.plan.physical.PhysicalPlan.<init>(PhysicalPlan.java:378) at com.datatorrent.stram.StreamingContainerManager.<init>(StreamingContainerManager.java:418) at com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingContainerManager.java:3023) at com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingAppMasterService.java:551) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:102) End of LogType:dt.log LogType:launch_container.sh Log Upload Time:Tue Dec 20 12:44:49 -0800 2016 LogLength:20175 Log Contents: #!/bin/bash Thanks and Regards, Bibin John| Data Movement Technology Development 20205 North Creek Pkwy , Bothell, WA 98011 USA • Office: (770) 235 5614 | Cell: (469) 648-9858 Email: [email protected]<mailto:[email protected]> From: Chaitanya Chebolu [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, December 20, 2016 2:51 AM To: [email protected]<mailto:[email protected]> Subject: Re: Property Name for Kafka config Hi John, Please refer the below example application: https://github.com/DataTorrent/examples/tree/master/tutorials/exactly-once and documentation of Kafka Input Operator at https://github.com/apache/apex-malhar/blob/master/docs/operators/kafkaInputOperator.md Regards, Chaitanya On Tue, Dec 20, 2016 at 3:47 PM, JOHN, BIBIN <[email protected]<mailto:[email protected]>> wrote: All, Could you please let me know name of the properties which I must use for below operator? KafkaSinglePortStringInputOperator
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> <modelVersion>4.0.0</modelVersion> <groupId>com.att.dti</groupId> <version>1.0-SNAPSHOT</version> <artifactId>jdbcToJdbc</artifactId> <packaging>jar</packaging> <!-- change these to the appropriate values --> <name>JDBC2Jdbcr</name> <description>Example Use of JDBC Input Operator</description> <properties> <!-- change this if you desire to use a different version of Apex Core --> <apex.version>3.4.0</apex.version> <apex.apppackage.classpath>lib/*.jar</apex.apppackage.classpath> <malhar.version>3.5.0</malhar.version> <apex.apppackage.classpath>lib/*.jar</apex.apppackage.classpath> <datatorrent.apppackage.classpath>lib/*.jar</datatorrent.apppackage.classpath> </properties> <repositories> <repository> <snapshots> <enabled>false</enabled> </snapshots> <id>Datatorrent-Releases</id> <name>DataTorrent Release Repository</name> <url>https://www.datatorrent.com/maven/content/repositories/releases/</url> </repository> </repositories> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-eclipse-plugin</artifactId> <version>2.9</version> <configuration> <downloadSources>true</downloadSources> </configuration> </plugin> <plugin> <artifactId>maven-compiler-plugin</artifactId> <version>3.3</version> <configuration> <encoding>UTF-8</encoding> <source>1.7</source> <target>1.7</target> <debug>true</debug> <optimize>false</optimize> <showDeprecation>true</showDeprecation> <showWarnings>true</showWarnings> </configuration> </plugin> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>2.8</version> <executions> <execution> <id>copy-dependencies</id> <phase>prepare-package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <outputDirectory>target/deps</outputDirectory> <includeScope>runtime</includeScope> </configuration> </execution> </executions> </plugin> <plugin> <artifactId>maven-assembly-plugin</artifactId> <executions> <execution> <id>app-package-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> <configuration> <finalName>${project.artifactId}-${project.version}-apexapp</finalName> <appendAssemblyId>false</appendAssemblyId> <descriptors> <descriptor>src/assemble/appPackage.xml</descriptor> </descriptors> <archiverConfig> <defaultDirectoryMode>0755</defaultDirectoryMode> </archiverConfig> <archive> <manifestEntries> <Class-Path>${apex.apppackage.classpath}</Class-Path> <DT-Engine-Version>${apex.version}</DT-Engine-Version> <DT-App-Package-Group-Id>${project.groupId}</DT-App-Package-Group-Id> <DT-App-Package-Name>${project.artifactId}</DT-App-Package-Name> <DT-App-Package-Version>${project.version}</DT-App-Package-Version> <DT-App-Package-Display-Name>${project.name}</DT-App-Package-Display-Name> <DT-App-Package-Description>${project.description}</DT-App-Package-Description> </manifestEntries> </archive> </configuration> </execution> </executions> </plugin> <plugin> <artifactId>maven-antrun-plugin</artifactId> <version>1.7</version> <executions> <execution> <phase>package</phase> <configuration> <target> <move file="${project.build.directory}/${project.artifactId}-${project.version}-apexapp.jar" tofile="${project.build.directory}/${project.artifactId}-${project.version}.apa" /> </target> </configuration> <goals> <goal>run</goal> </goals> </execution> <execution> <!-- create resource directory for xml javadoc --> <id>createJavadocDirectory</id> <phase>generate-resources</phase> <configuration> <tasks> <delete dir="${project.build.directory}/generated-resources/xml-javadoc" /> <mkdir dir="${project.build.directory}/generated-resources/xml-javadoc" /> </tasks> </configuration> <goals> <goal>run</goal> </goals> </execution> </executions> </plugin> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>1.9.1</version> <executions> <execution> <id>attach-artifacts</id> <phase>package</phase> <goals> <goal>attach-artifact</goal> </goals> <configuration> <artifacts> <artifact> <file>target/${project.artifactId}-${project.version}.apa</file> <type>apa</type> </artifact> </artifacts> <skipAttach>false</skipAttach> </configuration> </execution> </executions> </plugin> <!-- generate javdoc --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-javadoc-plugin</artifactId> <executions> <!-- generate xml javadoc --> <execution> <id>xml-doclet</id> <phase>generate-resources</phase> <goals> <goal>javadoc</goal> </goals> <configuration> <doclet>com.github.markusbernhardt.xmldoclet.XmlDoclet</doclet> <additionalparam>-d ${project.build.directory}/generated-resources/xml-javadoc -filename ${project.artifactId}-${project.version}-javadoc.xml</additionalparam> <useStandardDocletOptions>false</useStandardDocletOptions> <docletArtifact> <groupId>com.github.markusbernhardt</groupId> <artifactId>xml-doclet</artifactId> <version>1.0.4</version> </docletArtifact> </configuration> </execution> </executions> </plugin> <!-- Transform xml javadoc to stripped down version containing only class/interface comments and tags --> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>xml-maven-plugin</artifactId> <version>1.0</version> <executions> <execution> <id>transform-xmljavadoc</id> <phase>generate-resources</phase> <goals> <goal>transform</goal> </goals> </execution> </executions> <configuration> <transformationSets> <transformationSet> <dir>${project.build.directory}/generated-resources/xml-javadoc</dir> <includes> <include>${project.artifactId}-${project.version}-javadoc.xml</include> </includes> <stylesheet>XmlJavadocCommentsExtractor.xsl</stylesheet> <outputDir>${project.build.directory}/generated-resources/xml-javadoc</outputDir> </transformationSet> </transformationSets> </configuration> </plugin> <!-- copy xml javadoc to class jar --> <plugin> <artifactId>maven-resources-plugin</artifactId> <version>2.6</version> <executions> <execution> <id>copy-resources</id> <phase>process-resources</phase> <goals> <goal>copy-resources</goal> </goals> <configuration> <outputDirectory>${basedir}/target/classes</outputDirectory> <resources> <resource> <directory>${project.build.directory}/generated-resources/xml-javadoc</directory> <includes> <include>${project.artifactId}-${project.version}-javadoc.xml</include> </includes> <filtering>true</filtering> </resource> </resources> </configuration> </execution> </executions> </plugin> </plugins> </build> <dependencies> <!-- add your dependencies here --> <dependency> <groupId>org.apache.apex</groupId> <artifactId>malhar-library</artifactId> <version>${malhar.version}</version> <!-- If you know that your application does not need transitive dependencies pulled in by malhar-library, uncomment the following to reduce the size of your app package. --> <exclusions> <exclusion> <groupId>*</groupId> <artifactId>*</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.apex</groupId> <artifactId>apex-common</artifactId> <version>${apex.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.10</version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.apex</groupId> <artifactId>apex-engine</artifactId> <version>${apex.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>com.oracle</groupId> <artifactId>ojdbc6</artifactId> <scope>compile</scope> <version>11.2.0.4</version> </dependency> <dependency> <groupId>org.codehaus.janino</groupId> <artifactId>janino</artifactId> <version>2.7.8</version> </dependency> <dependency> <groupId>org.codehaus.janino</groupId> <artifactId>commons-compiler</artifactId> <version>2.7.8</version> </dependency> </dependencies> </project>
