Hi Srinath, I think Tharinda meant to say that he is in the process of running a Spark job. Tharinda, isn't it the case?
On Tue, Mar 17, 2015 at 7:39 AM, Srinath Perera <[email protected]> wrote: > Why do you need to build it? ( it can be built), but you can use the jars. > > --Srinth > > On Mon, Mar 16, 2015 at 3:30 PM, Tharinda Ehelepola <[email protected] > > wrote: > >> Hi, >> >> I studied about apache spark these days. I 'm still stuck in building >> part. I went through different links and finally decided to build it using >> intellij. I have found most of the libraries but still errors occur when >> running examples. >> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/commons/lang3/SystemUtils >> at org.apache.spark.util.Utils$.<init>(Utils.scala:1476) >> at org.apache.spark.util.Utils$.<clinit>(Utils.scala) >> at org.apache.spark.SparkContext.<init>(SparkContext.scala:70) >> at >> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) >> at org.apache.spark.examples.JavaLogQuery.main(JavaLogQuery.java:105) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:483) >> at >> com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.commons.lang3.SystemUtils >> at java.net.URLClassLoader$1.run(URLClassLoader.java:372) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:360) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> ... 10 more >> Exception in thread "delete Spark temp dirs" >> java.lang.NoClassDefFoundError: Could not initialize class >> org.apache.spark.util.Utils$ >> at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:175) >> >> >> Here I added some errors. *Apache Spark* is a fast and general engine >> for big data processing. Wrangler is to data cleaning and transformation. >> So what I got is some spark methods related to wrangler scripts. If I can >> get proper knowledge about spark java API I can use that for conversion of >> wrangler scripts to java. And also it will help me to get proper idea about >> the whole project. So can you please give me some tips for building apache >> spark. >> >> Thank you, >> Tharinda. >> >> On Fri, Mar 13, 2015 at 12:24 PM, Tharinda Ehelepola < >> [email protected]> wrote: >> >>> Hi, >>> >>> I got the libraries. But still have a problem with below ones. >>> >>> import org.apache.spark.streaming.api.java.JavaDStream; >>> import org.apache.spark.streaming.api.java.JavaReceiverInputDStream; >>> >>> import org.apache.spark.streaming.api.java.JavaStreamingContext; >>> >>> >>> On Fri, Mar 13, 2015 at 5:05 AM, Supun Sethunga <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> You can find all the necessary libraries in the Maven Repository [1]. >>>> Please use the latest version of spark (v1.2.1). >>>> >>>> [1] http://mvnrepository.com/artifact/org.apache.spark >>>> >>>> Regards, >>>> Supun >>>> >>>> On Fri, Mar 13, 2015 at 6:27 AM, Tharinda Ehelepola < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I checked the methods of apache spark. Some of them are related to >>>>> wrangler script. I run spark as a maven project. But I am not able to run >>>>> example code in java. Because it asking about the libraries. I found >>>>> some simple example from stackoverflow . But I am not able to find the >>>>> library of spark.apache. So can you please give me some idea about >>>>> those libraries. >>>>> >>>>> On Thu, Mar 12, 2015 at 11:35 AM, Supun Sethunga <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Tharinda, >>>>>> >>>>>> Please find the comments inline. >>>>>> >>>>>> I think the major part of this project is create a back end of java >>>>>>> code created using wrangler script. >>>>>> >>>>>> Yes. >>>>>> >>>>>> Wrangler scripts are mostly done by javascript. So I need to convert >>>>>>> those into java. >>>>>> >>>>>> Yes. Further, java code should be Apache Spark Transformation. Please >>>>>> refer my previous mail for details on Spark. >>>>>> >>>>>> I was not able to find some method. Other ways of doing this is used >>>>>>> common methods like split,merge.. etc easily find on java. Rest of the >>>>>>> methods are convert by manually. Is there any method to convert those >>>>>>> scripts to java. >>>>>> >>>>>> You have to come up with a solution and an implementation for this. >>>>>> It is the core part of the project :) >>>>>> >>>>>> Feel free to contact us if you need further clarifications. >>>>>> >>>>>> Regards, >>>>>> Supun >>>>>> >>>>>> On Thu, Mar 12, 2015 at 1:04 PM, Tharinda Ehelepola < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I think the major part of this project is create a back end of java >>>>>>> code created using wrangler script. Wrangler scripts are mostly done by >>>>>>> javascript. So I need to convert those into java. I was not able to find >>>>>>> some method. Other ways of doing this is used common methods like >>>>>>> split,merge.. etc easily find on java. Rest of the methods are convert >>>>>>> by >>>>>>> manually. Is there any method to convert those scripts to java. >>>>>>> >>>>>>> Thank you. >>>>>>> Tharinda. >>>>>>> >>>>>>> On Wed, Mar 11, 2015 at 3:45 AM, Supun Sethunga <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Tharinda, >>>>>>>> >>>>>>>> Ultimately, the java code created using the wrangler script (after >>>>>>>> conversion), should be able to used as an Apache Spark [1] >>>>>>>> Transformations >>>>>>>> [2]. (at the back-end) >>>>>>>> >>>>>>>> On a side note, please use the WSO2 dev mailing list for any >>>>>>>> project related communication :) >>>>>>>> >>>>>>>> [1] https://spark.apache.org/ >>>>>>>> [2] >>>>>>>> http://spark.apache.org/docs/1.2.0/programming-guide.html#transformations >>>>>>>> >>>>>>>> Regards, >>>>>>>> Supun >>>>>>>> >>>>>>>> On Wed, Mar 11, 2015 at 7:52 AM, Nirmal Fernando <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes, that's exactly what we want ! >>>>>>>>> >>>>>>>>> On Wed, Mar 11, 2015 at 3:30 AM, Tharinda Ehelepola < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> I checked the libraries on wrangler. I think that you mentioned >>>>>>>>>> can be done using the libraries. What I think is get the libraries >>>>>>>>>> from >>>>>>>>>> wrangler and create a back-end for the interface created using >>>>>>>>>> jaggery. >>>>>>>>>> Wrangler script to java converter can be add to the back-end also. >>>>>>>>>> >>>>>>>>>> On Tue, Mar 10, 2015 at 4:10 AM, Nirmal Fernando <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Remotely connecting to Wrangler (hosted service) is not a good >>>>>>>>>>> option. What we expect is we take their library (it's Open source) >>>>>>>>>>> and use >>>>>>>>>>> the API to perform the transformations and cleaning. Could you >>>>>>>>>>> please check >>>>>>>>>>> the feasibility of that? >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 10, 2015 at 9:34 AM, Tharinda Ehelepola < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I drew what I understood the project to be. >>>>>>>>>>>> >>>>>>>>>>>> If one wants to do data cleaning and transformation he need to >>>>>>>>>>>> connect to the wrangler directly. That is what I mean from the >>>>>>>>>>>> first two >>>>>>>>>>>> blocks. If they have a customized UI you can do data cleaning >>>>>>>>>>>> and transformation using ML also. They don't need to connect to the >>>>>>>>>>>> wrangler. Another thing i realized that is they don't need all the >>>>>>>>>>>> operations in wrangler. They need some specific subset of >>>>>>>>>>>> operations only. >>>>>>>>>>>> What I mean from the bottom block is the UI created using Jaggery >>>>>>>>>>>> interfacing between ML and the wrangler. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Mar 10, 2015 at 2:16 AM, Nirmal Fernando < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Tharindu, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the diagram. Do you mind explaining the diagram? >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Mar 10, 2015 at 3:27 AM, Tharinda Ehelepola < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here I attached a simple design of project development. Can >>>>>>>>>>>>>> you give me some comments about this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Mar 9, 2015 at 2:16 AM, Nirmal Fernando < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Our ultimate goal is to provide data cleaning and >>>>>>>>>>>>>>> transformation. Hence, Tharindu, you need to pick the things >>>>>>>>>>>>>>> that is >>>>>>>>>>>>>>> sufficient to perform those operations. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What we prefer is a D3 based UI. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sun, Mar 8, 2015 at 10:34 PM, Tharinda Ehelepola < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Wrangler already has a UI. I think that you excepting >>>>>>>>>>>>>>>> customized >>>>>>>>>>>>>>>> UI for Wrangler using Jaggery. Which kind of conditions and >>>>>>>>>>>>>>>> options do you >>>>>>>>>>>>>>>> excepting from that UI. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 2:13 PM, Tharinda Ehelepola < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for your response. I have some idea about both Data >>>>>>>>>>>>>>>>> Wrangler API and Jaggery. I'll try to bulid a UI. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 11:04 AM, Nirmal Fernando < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 4:34 PM, Nirmal Fernando < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Tharinda, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks for your interest on the proposal #17. Actually, >>>>>>>>>>>>>>>>>>> it is not for WSO2 CEP but for WSO2 Machine Learner which >>>>>>>>>>>>>>>>>>> is a new product >>>>>>>>>>>>>>>>>>> that we are building these days. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The main idea of this project is to integrate the Data >>>>>>>>>>>>>>>>>>> Wrangler to WSO2 ML and get the data cleaning, >>>>>>>>>>>>>>>>>>> transformation features into >>>>>>>>>>>>>>>>>>> WSO2 ML. But as a start, you could play with Data Wrangler >>>>>>>>>>>>>>>>>>> API and check >>>>>>>>>>>>>>>>>>> how you could build a UI (preferably using HTML, Jaggery) >>>>>>>>>>>>>>>>>>> that exposes Data >>>>>>>>>>>>>>>>>>> Wrangler APIs. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Sat, Mar 7, 2015 at 2:53 PM, Tharinda Ehelepola < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I am a student of university of Peradeniya, Faculty of >>>>>>>>>>>>>>>>>>>> engineering, Department of computer engineering. I am >>>>>>>>>>>>>>>>>>>> interested about your >>>>>>>>>>>>>>>>>>>> projects. I have already seen the >>>>>>>>>>>>>>>>>>>> "Proposal 17 : Data Wrangler extension for WSO2 Machine >>>>>>>>>>>>>>>>>>>> Learner". Actually i am very interested about that >>>>>>>>>>>>>>>>>>>> project. I have already >>>>>>>>>>>>>>>>>>>> install the CEP. Now i really want to see some sample >>>>>>>>>>>>>>>>>>>> extensions. I found >>>>>>>>>>>>>>>>>>>> the link but Google drive not give the permission for me. >>>>>>>>>>>>>>>>>>>> Could you please >>>>>>>>>>>>>>>>>>>> give me some example extensions or approval for the Google >>>>>>>>>>>>>>>>>>>> drive. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thank You. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, >>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, >>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 >>>>>>>>>>>>>>> Inc. >>>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>> Nirmal >>>>>>>>>>>>> >>>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Thanks & regards, >>>>>>>>>>> Nirmal >>>>>>>>>>> >>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks & regards, >>>>>>>>> Nirmal >>>>>>>>> >>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>> Mobile: +94715779733 >>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Supun Sethunga* >>>>>>>> Software Engineer >>>>>>>> WSO2, Inc. >>>>>>>> lean | enterprise | middleware >>>>>>>> Mobile : +94 716546324 >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Supun Sethunga* >>>>>> Software Engineer >>>>>> WSO2, Inc. >>>>>> http://wso2.com/ >>>>>> lean | enterprise | middleware >>>>>> Mobile : +94 716546324 >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Supun Sethunga* >>>> Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> >>> >>> >> > > > -- > ============================ > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > -- Thanks & regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
