Yes, I have get the same error, and can't resolve it.
2016-02-05 0:29 GMT+08:00 K Zharas <[email protected]>:
> Thank you for your reply, it worked. However, I got an another problem.
>
> Basically, I'm trying to implement HDFSBolt in Storm. I wanted to start
> with basic one, so used TestWordSpout provided by Storm.
>
> I can successfully compile and submit the topology, but it doesn't write
> into HDFS.
>
> In StormUI, I can see that Spout is emitting continuously. Bolt doesn't
> do anything, and it has an error
>
> java.lang.NoClassDefFoundError: org/apache/hadoop/fs/CanUnbuffer at
> java.lang.ClassLoader.defineClass1(Native Method) at
> java.lang.ClassLoader.defineClass(ClassLoader.java:800) at
> java.security.Sec
>
> Here is my topology
>
> public class HdfsFileTopology {
> public static void main(String[] args) throws Exception {
> RecordFormat format = new DelimitedRecordFormat().withFieldDelimiter(",");
> SyncPolicy syncPolicy = new CountSyncPolicy(100);
> FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(10.0f,
> Units.KB);
> FileNameFormat fileNameFormat = new
> DefaultFileNameFormat().withPath("/user");
> HdfsBolt bolt = new HdfsBolt()
> .withFsUrl("hdfs://localhost:9000")
> .withFileNameFormat(fileNameFormat)
> .withRecordFormat(format)
> .withRotationPolicy(rotationPolicy)
> .withSyncPolicy(syncPolicy);
>
> TopologyBuilder builder = new TopologyBuilder();
> builder.setSpout("word", new TestWordSpout(), 1);
> builder.setBolt("output", bolt, 1).shuffleGrouping("word");
> Config conf = new Config();
> conf.setDebug(true);
> conf.setNumWorkers(3);
> StormSubmitter.submitTopology("HdfsFileTopology", conf,
> builder.createTopology());
> }
> }
>
>
> On Thu, Feb 4, 2016 at 5:04 AM, P. Taylor Goetz <[email protected]> wrote:
>
>> Assuming you have git and maven installed:
>>
>> git clone [email protected]:apache/storm.git
>> cd storm
>> git checkout -b 1.x origin/1.x-branch
>> mvn install -DskipTests
>>
>> That third step checks out the 1.x-branch branch which is the base for
>> the upcoming 1.0 release.
>>
>> You can then include the storm-hdfs dependency in your project:
>>
>> <dependency>
>> <groupId>org.apache.storm</groupId>
>> <artifactId>storm-hdfs</artifactId>
>> <version>1.0.0-SNAPSHOT</version>
>> </dependency>
>>
>> You can find more information on using the spout and other HDFS
>> components here:
>>
>>
>> https://github.com/apache/storm/tree/1.x-branch/external/storm-hdfs#hdfs-spout
>>
>> -Taylor
>>
>> On Feb 3, 2016, at 2:54 PM, K Zharas <[email protected]> wrote:
>>
>> Oh ok. Can you plz give me an idea how can I do it manually? I'm quite
>> beginner :)
>>
>> On Thu, Feb 4, 2016 at 3:43 AM, Parth Brahmbhatt <
>> [email protected]> wrote:
>>
>>> Storm-hdfs spout is not yet published in maven. You will have to
>>> checkout storm locally and build it to make it available for development.
>>>
>>> From: K Zharas <[email protected]>
>>> Reply-To: "[email protected]" <[email protected]>
>>> Date: Wednesday, February 3, 2016 at 11:41 AM
>>> To: "[email protected]" <[email protected]>
>>> Subject: Re: Storm + HDFS
>>>
>>> Yes, looks like it is. But, I have added dependencies required by
>>> storm-hdfs as stated in a guide.
>>>
>>> On Thu, Feb 4, 2016 at 3:33 AM, Nick R. Katsipoulakis <
>>> [email protected]> wrote:
>>>
>>>> Well,
>>>>
>>>> those errors look like a problem with the way you build your jar file.
>>>> Please, make sure that you build your jar with the proper storm maven
>>>> dependency).
>>>>
>>>> Cheers,
>>>> Nick
>>>>
>>>> On Wed, Feb 3, 2016 at 2:31 PM, K Zharas <[email protected]> wrote:
>>>>
>>>>> It throws and error that packages does not exist. I have also tried
>>>>> changing org.apache to backtype, still got an error but only for
>>>>> storm.hdfs.spout. Btw, I use Storm-0.10.0 and Hadoop-2.7.1
>>>>>
>>>>> package org.apache.storm does not exist
>>>>> package org.apache.storm does not exist
>>>>> package org.apache.storm.generated does not exist
>>>>> package org.apache.storm.metric does not exist
>>>>> package org.apache.storm.topology does not exist
>>>>> package org.apache.storm.utils does not exist
>>>>> package org.apache.storm.utils does not exist
>>>>> package org.apache.storm.hdfs.spout does not exist
>>>>> package org.apache.storm.hdfs.spout does not exist
>>>>> package org.apache.storm.topology.base does not exist
>>>>> package org.apache.storm.topology does not exist
>>>>> package org.apache.storm.tuple does not exist
>>>>> package org.apache.storm.task does not exist
>>>>>
>>>>> On Wed, Feb 3, 2016 at 8:57 PM, Matthias J. Sax <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Storm does provide HdfsSpout and HdfsBolt already. Just use those,
>>>>>> instead of writing your own spout/bolt:
>>>>>>
>>>>>> https://github.com/apache/storm/tree/master/external/storm-hdfs
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>> On 02/03/2016 12:34 PM, K Zharas wrote:
>>>>>> > Can anyone help to create a Spout which reads a file from HDFS?
>>>>>> > I have tried with the code below, but it is not working.
>>>>>> >
>>>>>> > public void nextTuple() {
>>>>>> > Path pt=new Path("hdfs://localhost:50070/user/BCpredict.txt
>>>>>> ");
>>>>>> > FileSystem fs = FileSystem.get(new Configuration());
>>>>>> > BufferedReader br = new BufferedReader(new
>>>>>> > InputStreamReader(fs.open(pt)));
>>>>>> > String line = br.readLine();
>>>>>> > while (line != null){
>>>>>> > System.out.println(line);
>>>>>> > line=br.readLine();
>>>>>> > _collector.emit(new Values(line));
>>>>>> > }
>>>>>> > }
>>>>>> >
>>>>>> > On Tue, Feb 2, 2016 at 1:19 PM, K Zharas <[email protected]
>>>>>> > <mailto:[email protected]>> wrote:
>>>>>> >
>>>>>> > Hi.
>>>>>> >
>>>>>> > I have a project I'm currently working on. The idea is to
>>>>>> implement
>>>>>> > "scikit-learn" into Storm and integrate it with HDFS.
>>>>>> >
>>>>>> > I've already implemented "scikit-learn". But, currently I'm
>>>>>> using a
>>>>>> > text file to read and write. However, I need to use HDFS, but
>>>>>> > finding it hard to integrate with HDFS.
>>>>>> >
>>>>>> > Here is the link to github
>>>>>> > <https://github.com/kgzharas/StormTopologyTest>. (I only
>>>>>> included
>>>>>> > files that I used, not whole project)
>>>>>> >
>>>>>> > Basically, I have a few questions if you don't mint to answer
>>>>>> them
>>>>>> > 1) How to use HDFS to read and write?
>>>>>> > 2) Is my "scikit-learn" implementation correct?
>>>>>> > 3) How to create a Storm project? (Currently working in
>>>>>> "storm-starter")
>>>>>> >
>>>>>> > These questions may sound a bit silly, but I really can't find a
>>>>>> > proper solution.
>>>>>> >
>>>>>> > Thank you for your attention to this matter.
>>>>>> > Sincerely, Zharas.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Best regards,
>>>>>> > Zharas
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Zharas
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Nick R. Katsipoulakis,
>>>> Department of Computer Science
>>>> University of Pittsburgh
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Zharas
>>>
>>
>>
>>
>> --
>> Best regards,
>> Zharas
>>
>>
>>
>
>
> --
> Best regards,
> Zharas
>
