Why do you need to build it? ( it can be built), but you can use the jars. --Srinth
On Mon, Mar 16, 2015 at 3:30 PM, Tharinda Ehelepola <[email protected]> wrote: > Hi, > > I studied about apache spark these days. I 'm still stuck in building > part. I went through different links and finally decided to build it using > intellij. I have found most of the libraries but still errors occur when > running examples. > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/commons/lang3/SystemUtils > at org.apache.spark.util.Utils$.<init>(Utils.scala:1476) > at org.apache.spark.util.Utils$.<clinit>(Utils.scala) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:70) > at > org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) > at org.apache.spark.examples.JavaLogQuery.main(JavaLogQuery.java:105) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.lang3.SystemUtils > at java.net.URLClassLoader$1.run(URLClassLoader.java:372) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 10 more > Exception in thread "delete Spark temp dirs" > java.lang.NoClassDefFoundError: Could not initialize class > org.apache.spark.util.Utils$ > at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:175) > > > Here I added some errors. *Apache Spark* is a fast and general engine for > big data processing. Wrangler is to data cleaning and transformation. So > what I got is some spark methods related to wrangler scripts. If I can get > proper knowledge about spark java API I can use that for conversion of > wrangler scripts to java. And also it will help me to get proper idea about > the whole project. So can you please give me some tips for building apache > spark. > > Thank you, > Tharinda. > > On Fri, Mar 13, 2015 at 12:24 PM, Tharinda Ehelepola < > [email protected]> wrote: > >> Hi, >> >> I got the libraries. But still have a problem with below ones. >> >> import org.apache.spark.streaming.api.java.JavaDStream; >> import org.apache.spark.streaming.api.java.JavaReceiverInputDStream; >> >> import org.apache.spark.streaming.api.java.JavaStreamingContext; >> >> >> On Fri, Mar 13, 2015 at 5:05 AM, Supun Sethunga <[email protected]> wrote: >> >>> Hi, >>> >>> You can find all the necessary libraries in the Maven Repository [1]. >>> Please use the latest version of spark (v1.2.1). >>> >>> [1] http://mvnrepository.com/artifact/org.apache.spark >>> >>> Regards, >>> Supun >>> >>> On Fri, Mar 13, 2015 at 6:27 AM, Tharinda Ehelepola < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> I checked the methods of apache spark. Some of them are related to >>>> wrangler script. I run spark as a maven project. But I am not able to run >>>> example code in java. Because it asking about the libraries. I found >>>> some simple example from stackoverflow . But I am not able to find the >>>> library of spark.apache. So can you please give me some idea about >>>> those libraries. >>>> >>>> On Thu, Mar 12, 2015 at 11:35 AM, Supun Sethunga <[email protected]> >>>> wrote: >>>> >>>>> Hi Tharinda, >>>>> >>>>> Please find the comments inline. >>>>> >>>>> I think the major part of this project is create a back end of java >>>>>> code created using wrangler script. >>>>> >>>>> Yes. >>>>> >>>>> Wrangler scripts are mostly done by javascript. So I need to convert >>>>>> those into java. >>>>> >>>>> Yes. Further, java code should be Apache Spark Transformation. Please >>>>> refer my previous mail for details on Spark. >>>>> >>>>> I was not able to find some method. Other ways of doing this is used >>>>>> common methods like split,merge.. etc easily find on java. Rest of the >>>>>> methods are convert by manually. Is there any method to convert those >>>>>> scripts to java. >>>>> >>>>> You have to come up with a solution and an implementation for this. It >>>>> is the core part of the project :) >>>>> >>>>> Feel free to contact us if you need further clarifications. >>>>> >>>>> Regards, >>>>> Supun >>>>> >>>>> On Thu, Mar 12, 2015 at 1:04 PM, Tharinda Ehelepola < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I think the major part of this project is create a back end of java >>>>>> code created using wrangler script. Wrangler scripts are mostly done by >>>>>> javascript. So I need to convert those into java. I was not able to find >>>>>> some method. Other ways of doing this is used common methods like >>>>>> split,merge.. etc easily find on java. Rest of the methods are convert by >>>>>> manually. Is there any method to convert those scripts to java. >>>>>> >>>>>> Thank you. >>>>>> Tharinda. >>>>>> >>>>>> On Wed, Mar 11, 2015 at 3:45 AM, Supun Sethunga <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Tharinda, >>>>>>> >>>>>>> Ultimately, the java code created using the wrangler script (after >>>>>>> conversion), should be able to used as an Apache Spark [1] >>>>>>> Transformations >>>>>>> [2]. (at the back-end) >>>>>>> >>>>>>> On a side note, please use the WSO2 dev mailing list for any project >>>>>>> related communication :) >>>>>>> >>>>>>> [1] https://spark.apache.org/ >>>>>>> [2] >>>>>>> http://spark.apache.org/docs/1.2.0/programming-guide.html#transformations >>>>>>> >>>>>>> Regards, >>>>>>> Supun >>>>>>> >>>>>>> On Wed, Mar 11, 2015 at 7:52 AM, Nirmal Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes, that's exactly what we want ! >>>>>>>> >>>>>>>> On Wed, Mar 11, 2015 at 3:30 AM, Tharinda Ehelepola < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> I checked the libraries on wrangler. I think that you mentioned >>>>>>>>> can be done using the libraries. What I think is get the libraries >>>>>>>>> from >>>>>>>>> wrangler and create a back-end for the interface created using >>>>>>>>> jaggery. >>>>>>>>> Wrangler script to java converter can be add to the back-end also. >>>>>>>>> >>>>>>>>> On Tue, Mar 10, 2015 at 4:10 AM, Nirmal Fernando <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Remotely connecting to Wrangler (hosted service) is not a good >>>>>>>>>> option. What we expect is we take their library (it's Open source) >>>>>>>>>> and use >>>>>>>>>> the API to perform the transformations and cleaning. Could you >>>>>>>>>> please check >>>>>>>>>> the feasibility of that? >>>>>>>>>> >>>>>>>>>> On Tue, Mar 10, 2015 at 9:34 AM, Tharinda Ehelepola < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I drew what I understood the project to be. >>>>>>>>>>> >>>>>>>>>>> If one wants to do data cleaning and transformation he need to >>>>>>>>>>> connect to the wrangler directly. That is what I mean from the >>>>>>>>>>> first two >>>>>>>>>>> blocks. If they have a customized UI you can do data cleaning >>>>>>>>>>> and transformation using ML also. They don't need to connect to the >>>>>>>>>>> wrangler. Another thing i realized that is they don't need all the >>>>>>>>>>> operations in wrangler. They need some specific subset of >>>>>>>>>>> operations only. >>>>>>>>>>> What I mean from the bottom block is the UI created using Jaggery >>>>>>>>>>> interfacing between ML and the wrangler. >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 10, 2015 at 2:16 AM, Nirmal Fernando < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Tharindu, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the diagram. Do you mind explaining the diagram? >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Mar 10, 2015 at 3:27 AM, Tharinda Ehelepola < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Here I attached a simple design of project development. Can >>>>>>>>>>>>> you give me some comments about this. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Mar 9, 2015 at 2:16 AM, Nirmal Fernando < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Our ultimate goal is to provide data cleaning and >>>>>>>>>>>>>> transformation. Hence, Tharindu, you need to pick the things >>>>>>>>>>>>>> that is >>>>>>>>>>>>>> sufficient to perform those operations. >>>>>>>>>>>>>> >>>>>>>>>>>>>> What we prefer is a D3 based UI. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Mar 8, 2015 at 10:34 PM, Tharinda Ehelepola < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Wrangler already has a UI. I think that you excepting customized >>>>>>>>>>>>>>> UI for Wrangler using Jaggery. Which kind of conditions and >>>>>>>>>>>>>>> options do you >>>>>>>>>>>>>>> excepting from that UI. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 2:13 PM, Tharinda Ehelepola < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for your response. I have some idea about both Data >>>>>>>>>>>>>>>> Wrangler API and Jaggery. I'll try to bulid a UI. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 11:04 AM, Nirmal Fernando < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 4:34 PM, Nirmal Fernando < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Tharinda, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for your interest on the proposal #17. Actually, >>>>>>>>>>>>>>>>>> it is not for WSO2 CEP but for WSO2 Machine Learner which is >>>>>>>>>>>>>>>>>> a new product >>>>>>>>>>>>>>>>>> that we are building these days. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The main idea of this project is to integrate the Data >>>>>>>>>>>>>>>>>> Wrangler to WSO2 ML and get the data cleaning, >>>>>>>>>>>>>>>>>> transformation features into >>>>>>>>>>>>>>>>>> WSO2 ML. But as a start, you could play with Data Wrangler >>>>>>>>>>>>>>>>>> API and check >>>>>>>>>>>>>>>>>> how you could build a UI (preferably using HTML, Jaggery) >>>>>>>>>>>>>>>>>> that exposes Data >>>>>>>>>>>>>>>>>> Wrangler APIs. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 2:53 PM, Tharinda Ehelepola < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I am a student of university of Peradeniya, Faculty of >>>>>>>>>>>>>>>>>>> engineering, Department of computer engineering. I am >>>>>>>>>>>>>>>>>>> interested about your >>>>>>>>>>>>>>>>>>> projects. I have already seen the >>>>>>>>>>>>>>>>>>> "Proposal 17 : Data Wrangler extension for WSO2 Machine >>>>>>>>>>>>>>>>>>> Learner". Actually i am very interested about that project. >>>>>>>>>>>>>>>>>>> I have already >>>>>>>>>>>>>>>>>>> install the CEP. Now i really want to see some sample >>>>>>>>>>>>>>>>>>> extensions. I found >>>>>>>>>>>>>>>>>>> the link but Google drive not give the permission for me. >>>>>>>>>>>>>>>>>>> Could you please >>>>>>>>>>>>>>>>>>> give me some example extensions or approval for the Google >>>>>>>>>>>>>>>>>>> drive. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thank You. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, >>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 >>>>>>>>>>>>>>>>> Inc. >>>>>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>> >>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 >>>>>>>>>>>>>> Inc. >>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>> Nirmal >>>>>>>>>>>> >>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks & regards, >>>>>>>>>> Nirmal >>>>>>>>>> >>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>>> Mobile: +94715779733 >>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks & regards, >>>>>>>> Nirmal >>>>>>>> >>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>> Mobile: +94715779733 >>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Supun Sethunga* >>>>>>> Software Engineer >>>>>>> WSO2, Inc. >>>>>>> lean | enterprise | middleware >>>>>>> Mobile : +94 716546324 >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> http://wso2.com/ >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 >>>>> >>>> >>>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> > -- ============================ Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
