Hi,
I studied about apache spark these days. I 'm still stuck in building part.
I went through different links and finally decided to build it using
intellij. I have found most of the libraries but still errors occur when
running examples.
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/lang3/SystemUtils
at org.apache.spark.util.Utils$.<init>(Utils.scala:1476)
at org.apache.spark.util.Utils$.<clinit>(Utils.scala)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:70)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at org.apache.spark.examples.JavaLogQuery.main(JavaLogQuery.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.lang3.SystemUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
Exception in thread "delete Spark temp dirs"
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.util.Utils$
at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:175)
Here I added some errors. *Apache Spark* is a fast and general engine for
big data processing. Wrangler is to data cleaning and transformation. So
what I got is some spark methods related to wrangler scripts. If I can get
proper knowledge about spark java API I can use that for conversion of
wrangler scripts to java. And also it will help me to get proper idea about
the whole project. So can you please give me some tips for building apache
spark.
Thank you,
Tharinda.
On Fri, Mar 13, 2015 at 12:24 PM, Tharinda Ehelepola <[email protected]>
wrote:
> Hi,
>
> I got the libraries. But still have a problem with below ones.
>
> import org.apache.spark.streaming.api.java.JavaDStream;
> import org.apache.spark.streaming.api.java.JavaReceiverInputDStream;
>
> import org.apache.spark.streaming.api.java.JavaStreamingContext;
>
>
> On Fri, Mar 13, 2015 at 5:05 AM, Supun Sethunga <[email protected]> wrote:
>
>> Hi,
>>
>> You can find all the necessary libraries in the Maven Repository [1].
>> Please use the latest version of spark (v1.2.1).
>>
>> [1] http://mvnrepository.com/artifact/org.apache.spark
>>
>> Regards,
>> Supun
>>
>> On Fri, Mar 13, 2015 at 6:27 AM, Tharinda Ehelepola <
>> [email protected]> wrote:
>>
>>> Hi,
>>>
>>> I checked the methods of apache spark. Some of them are related to
>>> wrangler script. I run spark as a maven project. But I am not able to run
>>> example code in java. Because it asking about the libraries. I found
>>> some simple example from stackoverflow . But I am not able to find the
>>> library of spark.apache. So can you please give me some idea about
>>> those libraries.
>>>
>>> On Thu, Mar 12, 2015 at 11:35 AM, Supun Sethunga <[email protected]>
>>> wrote:
>>>
>>>> Hi Tharinda,
>>>>
>>>> Please find the comments inline.
>>>>
>>>> I think the major part of this project is create a back end of java
>>>>> code created using wrangler script.
>>>>
>>>> Yes.
>>>>
>>>> Wrangler scripts are mostly done by javascript. So I need to convert
>>>>> those into java.
>>>>
>>>> Yes. Further, java code should be Apache Spark Transformation. Please
>>>> refer my previous mail for details on Spark.
>>>>
>>>> I was not able to find some method. Other ways of doing this is used
>>>>> common methods like split,merge.. etc easily find on java. Rest of the
>>>>> methods are convert by manually. Is there any method to convert those
>>>>> scripts to java.
>>>>
>>>> You have to come up with a solution and an implementation for this. It
>>>> is the core part of the project :)
>>>>
>>>> Feel free to contact us if you need further clarifications.
>>>>
>>>> Regards,
>>>> Supun
>>>>
>>>> On Thu, Mar 12, 2015 at 1:04 PM, Tharinda Ehelepola <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I think the major part of this project is create a back end of java
>>>>> code created using wrangler script. Wrangler scripts are mostly done by
>>>>> javascript. So I need to convert those into java. I was not able to find
>>>>> some method. Other ways of doing this is used common methods like
>>>>> split,merge.. etc easily find on java. Rest of the methods are convert by
>>>>> manually. Is there any method to convert those scripts to java.
>>>>>
>>>>> Thank you.
>>>>> Tharinda.
>>>>>
>>>>> On Wed, Mar 11, 2015 at 3:45 AM, Supun Sethunga <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi Tharinda,
>>>>>>
>>>>>> Ultimately, the java code created using the wrangler script (after
>>>>>> conversion), should be able to used as an Apache Spark [1]
>>>>>> Transformations
>>>>>> [2]. (at the back-end)
>>>>>>
>>>>>> On a side note, please use the WSO2 dev mailing list for any project
>>>>>> related communication :)
>>>>>>
>>>>>> [1] https://spark.apache.org/
>>>>>> [2]
>>>>>> http://spark.apache.org/docs/1.2.0/programming-guide.html#transformations
>>>>>>
>>>>>> Regards,
>>>>>> Supun
>>>>>>
>>>>>> On Wed, Mar 11, 2015 at 7:52 AM, Nirmal Fernando <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, that's exactly what we want !
>>>>>>>
>>>>>>> On Wed, Mar 11, 2015 at 3:30 AM, Tharinda Ehelepola <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> I checked the libraries on wrangler. I think that you mentioned can
>>>>>>>> be done using the libraries. What I think is get the libraries from
>>>>>>>> wrangler and create a back-end for the interface created using jaggery.
>>>>>>>> Wrangler script to java converter can be add to the back-end also.
>>>>>>>>
>>>>>>>> On Tue, Mar 10, 2015 at 4:10 AM, Nirmal Fernando <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Remotely connecting to Wrangler (hosted service) is not a good
>>>>>>>>> option. What we expect is we take their library (it's Open source)
>>>>>>>>> and use
>>>>>>>>> the API to perform the transformations and cleaning. Could you please
>>>>>>>>> check
>>>>>>>>> the feasibility of that?
>>>>>>>>>
>>>>>>>>> On Tue, Mar 10, 2015 at 9:34 AM, Tharinda Ehelepola <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I drew what I understood the project to be.
>>>>>>>>>>
>>>>>>>>>> If one wants to do data cleaning and transformation he need to
>>>>>>>>>> connect to the wrangler directly. That is what I mean from the first
>>>>>>>>>> two
>>>>>>>>>> blocks. If they have a customized UI you can do data cleaning
>>>>>>>>>> and transformation using ML also. They don't need to connect to the
>>>>>>>>>> wrangler. Another thing i realized that is they don't need all the
>>>>>>>>>> operations in wrangler. They need some specific subset of operations
>>>>>>>>>> only.
>>>>>>>>>> What I mean from the bottom block is the UI created using Jaggery
>>>>>>>>>> interfacing between ML and the wrangler.
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 10, 2015 at 2:16 AM, Nirmal Fernando <[email protected]
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Tharindu,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the diagram. Do you mind explaining the diagram?
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 10, 2015 at 3:27 AM, Tharinda Ehelepola <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Here I attached a simple design of project development. Can you
>>>>>>>>>>>> give me some comments about this.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Mar 9, 2015 at 2:16 AM, Nirmal Fernando <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Our ultimate goal is to provide data cleaning and
>>>>>>>>>>>>> transformation. Hence, Tharindu, you need to pick the things that
>>>>>>>>>>>>> is
>>>>>>>>>>>>> sufficient to perform those operations.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What we prefer is a D3 based UI.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Mar 8, 2015 at 10:34 PM, Tharinda Ehelepola <
>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Wrangler already has a UI. I think that you excepting customized
>>>>>>>>>>>>>> UI for Wrangler using Jaggery. Which kind of conditions and
>>>>>>>>>>>>>> options do you
>>>>>>>>>>>>>> excepting from that UI.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 2:13 PM, Tharinda Ehelepola <
>>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for your response. I have some idea about both Data
>>>>>>>>>>>>>>> Wrangler API and Jaggery. I'll try to bulid a UI.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 11:04 AM, Nirmal Fernando <
>>>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 4:34 PM, Nirmal Fernando <
>>>>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Tharinda,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for your interest on the proposal #17. Actually, it
>>>>>>>>>>>>>>>>> is not for WSO2 CEP but for WSO2 Machine Learner which is a
>>>>>>>>>>>>>>>>> new product
>>>>>>>>>>>>>>>>> that we are building these days.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The main idea of this project is to integrate the Data
>>>>>>>>>>>>>>>>> Wrangler to WSO2 ML and get the data cleaning, transformation
>>>>>>>>>>>>>>>>> features into
>>>>>>>>>>>>>>>>> WSO2 ML. But as a start, you could play with Data Wrangler
>>>>>>>>>>>>>>>>> API and check
>>>>>>>>>>>>>>>>> how you could build a UI (preferably using HTML, Jaggery)
>>>>>>>>>>>>>>>>> that exposes Data
>>>>>>>>>>>>>>>>> Wrangler APIs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 2:53 PM, Tharinda Ehelepola <
>>>>>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am a student of university of Peradeniya, Faculty of
>>>>>>>>>>>>>>>>>> engineering, Department of computer engineering. I am
>>>>>>>>>>>>>>>>>> interested about your
>>>>>>>>>>>>>>>>>> projects. I have already seen the
>>>>>>>>>>>>>>>>>> "Proposal 17 : Data Wrangler extension for WSO2 Machine
>>>>>>>>>>>>>>>>>> Learner". Actually i am very interested about that project.
>>>>>>>>>>>>>>>>>> I have already
>>>>>>>>>>>>>>>>>> install the CEP. Now i really want to see some sample
>>>>>>>>>>>>>>>>>> extensions. I found
>>>>>>>>>>>>>>>>>> the link but Google drive not give the permission for me.
>>>>>>>>>>>>>>>>>> Could you please
>>>>>>>>>>>>>>>>>> give me some example extensions or approval for the Google
>>>>>>>>>>>>>>>>>> drive.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thank You.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> Dev mailing list
>>>>>>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks & regards,
>>>>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2
>>>>>>>>>>>>>>>>> Inc.
>>>>>>>>>>>>>>>>> Mobile: +94715779733
>>>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks & regards,
>>>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2
>>>>>>>>>>>>>>>> Inc.
>>>>>>>>>>>>>>>> Mobile: +94715779733
>>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks & regards,
>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>
>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc.
>>>>>>>>>>>>> Mobile: +94715779733
>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Thanks & regards,
>>>>>>>>>>> Nirmal
>>>>>>>>>>>
>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc.
>>>>>>>>>>> Mobile: +94715779733
>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Thanks & regards,
>>>>>>>>> Nirmal
>>>>>>>>>
>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc.
>>>>>>>>> Mobile: +94715779733
>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Nirmal
>>>>>>>
>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc.
>>>>>>> Mobile: +94715779733
>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Supun Sethunga*
>>>>>> Software Engineer
>>>>>> WSO2, Inc.
>>>>>> lean | enterprise | middleware
>>>>>> Mobile : +94 716546324
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Supun Sethunga*
>>>> Software Engineer
>>>> WSO2, Inc.
>>>> http://wso2.com/
>>>> lean | enterprise | middleware
>>>> Mobile : +94 716546324
>>>>
>>>
>>>
>>
>>
>> --
>> *Supun Sethunga*
>> Software Engineer
>> WSO2, Inc.
>> http://wso2.com/
>> lean | enterprise | middleware
>> Mobile : +94 716546324
>>
>
>
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
