That has solved the issue of importing the KafkaBolt class. Thank you!I see now
that i will need to include any dependencies into the pom filebefore i can use
any classes, etc.. from them.
I am getting a different error now, but after 2 days of no progress, newerrors
are good :-)
I am trying to use the example from the storm-kafka git account which is:
Note: I am using the ExclamationTopology to get all the components
workingbefore i begin to write my own topologies. It worked fine before i
started addingin the KafkaBolt.
public static void main(String[] args) throws Exception { TopologyBuilder
builder = new TopologyBuilder();
builder.setSpout("word", new MySQLSpout(), 10);
builder.setBolt("exclaim1", new ExclamationBolt(), 3).shuffleGrouping("word");
builder.setBolt("exclaim2", new ExclamationBolt(),
2).shuffleGrouping("exclaim1");
Config conf = new Config(); conf.setDebug(true);
//set producer properties. Properties props = new Properties();
props.put("bootstrap.servers", "192.168.1.23:9092"); props.put("acks", "1");
props.put("key.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer",
"org.apache.kafka.common.serialization.StringSerializer");
KafkaBolt bolt = new KafkaBolt();
bolt.withTopicSelector(new DefaultTopicSelector("test"));
bolt.withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper());
bolt.withProducerProperties(props);
builder.setBolt("forwardToKafka", bolt, 8).shuffleGrouping("exclaim2");
The error i am getting is regard to the bolt.withProducerProperties(props);
line.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile)
on project storm-starter: Compilation failure[ERROR]
/root/storm-starter/src/jvm/storm/starter/ExclamationTopology.java:[76,8]
error: cannot find symbol[ERROR] -> [Help
1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
(default-compile) on project storm-starter: Compilation
failure/root/storm-starter/src/jvm/storm/starter/ExclamationTopology.java:[76,8]
error: cannot find symbol
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at
org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at
org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at
org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)Caused
by: org.apache.maven.plugin.CompilationFailureException: Compilation
failure/root/storm-starter/src/jvm/storm/starter/ExclamationTopology.java:[76,8]
error: cannot find symbol
at
org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
at org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114) at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more[ERROR][ERROR] Re-run Maven using the -X switch to enable full
debug logging.[ERROR][ERROR] For more information about the errors and possible
solutions, please read the following articles:[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
I have checked and that is definitely the correct method name from the
KafkaBolt class.I have also tried using a different Properties object as the
parameter just in case that wasthe problem, but no luck. Again, any help will
be greatly appreciated.
ThanksDavid
> Subject: Re: Storm Kafka Bolt Integration
> To: [email protected]
> From: [email protected]
> Date: Sun, 21 Feb 2016 15:19:46 +0100
>
> Eclipse is not required. Just need to know what your setup is. :)
>
> Try to include storm-kafka as dependency:
>
> > <dependency>
> > <groupId>org.apache.storm</groupId>
> > <artifactId>storm-kafka</artifactId>
> > <version>0.9.0.1</version>
> > </dependency>
>
> Btw: I guess that your Storm dependency is not correct. I guess it
> should be:
>
> > <dependency>
> > <groupId>org.apache.storm</groupId>
> > <artifactId>storm-core</artifactId>
> > <version>0.9.0.1</version>
> > <!-- keep storm out of the jar-with-dependencies -->
> > <scope>provided</scope>
> > </dependency>
>
>
> -Matthias
>
>
> On 02/21/2016 03:09 PM, david kavanagh wrote:
> > Hi Matthias,
> >
> > I am using Maven as a build tool. I am not using any IDE. I have a Storm
> > cluster deployed on Ubuntu VMs,
> > and i am suing sshfs to mount my Storm folder from the remote machine,
> > to my local machine. Then
> > I am just using Sublime Text on my local machine to edit the necessary
> > files. Storm is working fine in both
> > local mode and as a cluster. It is just the KafkaBolt that is the problem.
> >
> > I am brand new to Storm/Maven/Kafka so i am hoping it is a small issue
> > that i am missing.
> > Here is the contents of my pom.xml
> >
> > /<project xmlns="http://maven.apache.org/POM/4.0.0";
> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/
> > / xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> > http://maven.apache.org/xsd/maven-4.0.0.xsd";>/
> > / <modelVersion>4.0.0</modelVersion>/
> > /
> > /
> > / <groupId>storm.starter</groupId>/
> > / <artifactId>storm-starter</artifactId>/
> > / <version>0.0.1-SNAPSHOT</version>/
> > / <packaging>jar</packaging>/
> > /
> > /
> > / <name>storm-starter</name>/
> > / <url>https://github.com/nathanmarz/storm-starter</url>/
> > /
> > /
> > / <properties>/
> > / <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>/
> > / </properties>/
> > /
> > /
> > / <repositories>/
> > / <repository>/
> > / <id>github-releases</id>/
> > /
> > <url>http://oss.sonatype.org/content/repositories/github-releases/</url>/
> > / </repository>/
> > / <repository>/
> > / <id>clojars.org</id>/
> > / <url>http://clojars.org/repo</url>/
> > / </repository>/
> > / </repositories>/
> > /
> > /
> > / <dependencies>/
> > / <dependency>/
> > / <groupId>org.apache.kafka</groupId>/
> > / <artifactId>kafka_2.9.2</artifactId>/
> > / <version>0.8.1.1</version>/
> > / <exclusions>/
> > / <exclusion>/
> > / <groupId>org.apache.zookeeper</groupId>/
> > / <artifactId>zookeeper</artifactId>/
> > / </exclusion>/
> > / <exclusion>/
> > / <groupId>log4j</groupId>/
> > / <artifactId>log4j</artifactId>/
> > / </exclusion>/
> > / </exclusions>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>mysql</groupId>/
> > / <artifactId>mysql-connector-java</artifactId>/
> > / <version>5.1.38</version>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>junit</groupId>/
> > / <artifactId>junit</artifactId>/
> > / <version>3.8.1</version>/
> > / <scope>test</scope>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>org.testng</groupId>/
> > / <artifactId>testng</artifactId>/
> > / <version>6.8.5</version>/
> > / <scope>test</scope>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>org.mockito</groupId>/
> > / <artifactId>mockito-all</artifactId>/
> > / <version>1.9.0</version>/
> > / <scope>test</scope>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>org.easytesting</groupId>/
> > / <artifactId>fest-assert-core</artifactId>/
> > / <version>2.0M8</version>/
> > / <scope>test</scope>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>org.jmock</groupId>/
> > / <artifactId>jmock</artifactId>/
> > / <version>2.6.0</version>/
> > / <scope>test</scope>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>storm</groupId>/
> > / <artifactId>storm</artifactId>/
> > / <version>0.9.0.1</version>/
> > / <!-- keep storm out of the jar-with-dependencies -->/
> > / <scope>provided</scope>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>commons-collections</groupId>/
> > / <artifactId>commons-collections</artifactId>/
> > / <version>3.2.1</version>/
> > / </dependency>/
> > / <dependency>/
> > / <groupId>com.google.guava</groupId>/
> > / <artifactId>guava</artifactId>/
> > / <version>15.0</version>/
> > / </dependency>/
> > / </dependencies>/
> > /
> > /
> > / <build>/
> > / <sourceDirectory>src/jvm</sourceDirectory>/
> > / <testSourceDirectory>test/jvm</testSourceDirectory>/
> > / <resources>/
> > / <resource>/
> > / <directory>${basedir}/multilang</directory>/
> > / </resource>/
> > / </resources>/
> > /
> > /
> > / <plugins>/
> > / <!--/
> > / Bind the maven-assembly-plugin to the package phase/
> > / this will create a jar file without the storm dependencies/
> > / suitable for deployment to a cluster./
> > / -->/
> > / <plugin>/
> > / <artifactId>maven-assembly-plugin</artifactId>/
> > / <configuration>/
> > / <descriptorRefs>/
> > / <descriptorRef>jar-with-dependencies</descriptorRef>/
> > / </descriptorRefs>/
> > / <archive>/
> > / <manifest>/
> > / <mainClass></mainClass>/
> > / </manifest>/
> > / </archive>/
> > / </configuration>/
> > / <executions>/
> > / <execution>/
> > / <id>make-assembly</id>/
> > / <phase>package</phase>/
> > / <goals>/
> > / <goal>single</goal>/
> > / </goals>/
> > / </execution>/
> > / </executions>/
> > / </plugin>/
> > / /
> > / <plugin>/
> > / <groupId>com.theoryinpractise</groupId>/
> > / <artifactId>clojure-maven-plugin</artifactId>/
> > / <version>1.3.12</version>/
> > / <extensions>true</extensions>/
> > / <configuration>/
> > / <sourceDirectories>/
> > / <sourceDirectory>src/clj</sourceDirectory>/
> > / </sourceDirectories>/
> > / </configuration>/
> > / <executions>/
> > / <execution>/
> > / <id>compile</id>/
> > / <phase>compile</phase>/
> > / <goals>/
> > / <goal>compile</goal>/
> > / </goals>/
> > / </execution>/
> > / <execution>/
> > / <id>test</id>/
> > / <phase>test</phase>/
> > / <goals>/
> > / <goal>test</goal>/
> > / </goals>/
> > / </execution>/
> > / </executions>/
> > / </plugin>/
> > /
> > /
> > / <plugin>/
> > / <groupId>org.codehaus.mojo</groupId>/
> > / <artifactId>exec-maven-plugin</artifactId>/
> > / <version>1.2.1</version>/
> > / <executions>/
> > / <execution>/
> > / <goals>/
> > / <goal>exec</goal>/
> > / </goals>/
> > / </execution>/
> > / </executions>/
> > / <configuration>/
> > / <executable>java</executable>/
> > / <includeProjectDependencies>true</includeProjectDependencies>/
> > / <includePluginDependencies>false</includePluginDependencies>/
> > / <classpathScope>compile</classpathScope>/
> > / <mainClass>${storm.topology}</mainClass>/
> > / </configuration>/
> > / </plugin>/
> > /
> > /
> > / <plugin>/
> > / <groupId>org.apache.maven.plugins</groupId>/
> > / <artifactId>maven-compiler-plugin</artifactId>/
> > / <configuration>/
> > / <source>1.7</source>/
> > / <target>1.7</target>/
> > / </configuration>/
> > / </plugin>/
> > /
> > /
> > / </plugins>/
> > / </build>/
> > /</project>/
> >
> > Any help at all would be really appreciated. I have been stuck on this
> > for 2 days now and i can't
> > really move forward with the project until i am sure i can get the
> > output from Storm into Kafka,
> > and in turn my application. I do have Eclipse on my local machine so i
> > can use that if it will help
> > figure out the issue. Thank you very much for your reply.
> >
> > Kind Regards
> > David
> >
> >
> >
> >
> >
> >
> >> Subject: Re: Storm Kafka Bolt Integration
> >> To: [email protected]
> >> From: [email protected]
> >> Date: Sun, 21 Feb 2016 14:44:00 +0100
> >>
> >> Hi David,
> >>
> >> yes you can just use the provide KafkaBolt.
> >>
> >> If the package is not found, I guess that you did not set up your
> >> development environment correctly. What do you use? An IDE? Some build
> >> tool like maven?
> >>
> >> -Matthias
> >>
> >> On 02/20/2016 02:13 PM, david kavanagh wrote:
> >> > Hello,
> >> >
> >> > I am using Storm as part of my fourth year project. I have a Storm
> >> > cluster running on some Ubuntu VM's
> >> > in an OpenStack environment. Kafka is setup and working fine on its own
> >> > VM. I am currently trying
> >> > to integrate the two so that a Storm Topology sends its output to a
> >> > Kafka Topic. I have been trying to understand
> >> > KafkaBolt but i am having difficulty implementing it.
> >> >
> >> > Reading the documentation
> >> > at http://storm.apache.org/documentation/storm-kafka.html , it seems to
> >> > me that
> >> > i can use a KafkaBolt class that has already been packaged. It is stated
> >> > in the documentation:
> >> >
> >> > "/You can create an instance of storm.kafka.bolt.KafkaBolt and attach it
> >> > as a component to your topology" /
> >> >
> >> > I have been trying to implement this but have had no success. No import
> >> > statement i have tried has worked.
> >> > i.e.
> >> > /
> >> > /
> >> > /import storm.kafka.bolt.KafkaBolt;/
> >> >
> >> > The error being thrown up is: /error: package storm.kafka.bolt
> >> > does not exist/
> >> >
> >> > I am new to Storm and Kafka and am aware that i may be getting the wrong
> >> > impression from the documentation.
> >> > Can i use import a KafkaBolt class or do i need to write my own
> >> > KafkaBolt class? My only reason for using Kafka is to
> >> > get the Storm Topology output to a Node.js application. I have searched
> >> > exhaustively online to find any information
> >> > on connecting Storms output to a node app and couldn't find anything
> >> > useful so Kafka seemed the only way to go.
> >> > If you could give me some information on sending a storm topologies
> >> > output to Kafka, or straight to a nodejs app
> >> > without Kafka, i would really appreciate it. Thank you in advance.
> >> >
> >> > Kind Regards
> >> > David Kavanagh
> >>
>
