Hi Srinath, "I think this is OK. I assume this is much easier. Let's do this and check the performance", I also think so. I am currently on this and have a progress in this.for your question, "If we use MOA directly, would we loose the distributed support in SAOMA. Let's do a call when you can, so we can dsicuss this in detail." I have to check for that.i think if we are using MOA, we can use distributed clusters. As i wen through their documentation MOA itself cannot support distribution. But samoa can. What samoa does is providing streaming and clustering support by wrapping MOA algorithms. So i think we dont need to go for that option,directly MOA. Because now we can handle samoa building blocks.So we had 2 options for integrating it with cep without exploiting samoa architecture which is highly modular,scalable and flexible. 1. Develop Samoa topologies with basic samoa building blocks which make use of MOA algorithms. 2. Creating New streaming options with samoa stream building blocks which can feed cep siddhi events into samoa streams and get results samoa streams to cep back.
As 2nd option is easy and take reasonable time i am currently developing some modules to integrate cep streams into samoa which can be easily further extended to 1 option as well. So i had to modify stream and entrance modules for that and i think is has good progress. So now i can feed my custom input stream to samoa topologies. That means i can easily integrate cep event stream into samoa instance stream. currently i am verifying the streaming clustering algorithms and its results with my custom input input streams which can be connected to samoa instance streams. As i have already developed siddhi extension for streaming, i can use them to feed my custom input streams now. As the initial step i am go with the streaming clustering algorithms. Those are in my GSOC github repo [1]. clustering - Streaming Clustering Support with samoa and CEP streaming - Streaming extension for samoa for cep evet streams i am currenlty working on the verification of results with some of our custom streams and then we will just have to integrate it with my extensions, which are already developed for cep as my first part.thank you. regards, Mahesh. [1] https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/samoa/streaming/src/main/java/org/gsoc/samoa/streaming On Fri, Jul 22, 2016 at 3:17 PM, Srinath Perera <[email protected]> wrote: > Hi Mahesh, > > On Thu, Jul 21, 2016 at 2:10 PM, Mahesh Dananjaya < > [email protected]> wrote: > >> Hi All, >> I am onto connecting cep streams with samoa streams to data analysis >> using samoa framework. To connect samoa with cep siddhi event streams what >> i we can do is that try to convert cep streams into samoa streams or else >> writing wrpper for samoa for cep streasm to be used. In both cases i have >> to covert siddhi cep streasm into samoa streams. Samoa is using MOA to >> analyse data. Moo contains ML framework to analyse stream data. Samoa is >> wrapping MOA withsome of its classes. >> >> Samoa streams is based on MOA, Instance and InstanceStreams. Samoa see >> streams as a stream of instances [1]. So if we are going to convert cep >> events into samoa instances , it will take time. But if we have some >> similarity between cep siddhi streams and samoa streasm we can reduce the >> time. >> 1. What is the underlying infrastructure for cep siddhi streasm.? >> 2. Are there anything as Instances or InstanceStreams kind of >> implmentation underlying cep streams? >> 3. How can i get more underestanding on CEP siddhi streams. >> >> On the other hand i can use my cep siddhi extension and put those events >> into event queue and convert them into samoa instances and feed them into >> samoa streaming ml topologies. >> > I think this is OK. I assume this is much easier. Let's do this and check > the performance. > > >> There is another option. In Samoa what they are basically doing is that >> wrapping MOA ML framework and write some classes for build streaming ml >> topologies. So as the other option i can wrap samoa moa with my design and >> use moa ml framework directly. (No need for Samoa extension). I have >> building some topologies to streaming data analysis [2]. Main problem is >> that lack of documentation. Anyway i had go through their whole samoa >> design.thank you. >> > > If we use MOA directly, would we loose the distributed support in SAOMA. > Let's do a call when you can, so we can dsicuss this in detail. > > --Srinath > > > >> regards, >> Mahesh. >> >> [1] >> https://github.com/apache/incubator-samoa/blob/master/samoa-api/src/main/java/org/apache/samoa/streams/clustering/ClusteringStream.java >> [2] >> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/samoa/streaming/src/main/java/org/gsoc/samoa/streaming >> >> On Mon, Jul 18, 2016 at 11:40 AM, Mahesh Dananjaya < >> [email protected]> wrote: >> >>> Hi Maheshakya, >>> Samoa modules built as topologies that connect streams with the internal >>> processors. I have already written some examples to test the ML algorithms >>> and samoa analysis topologies. What we need to done is mostly developing a >>> wrapper around samoa topologies to connect their input and output streams >>> with our cep streams. So i am currently going through their stream >>> architecture to connect our streams with their streams. Couple of examples >>> exapaining samoa ml topologies and streaming can be found in my git hub >>> repo [1]. Samoa using MOA ml algorithms by wrapping them with their >>> classes. Initailly i am trying to develop a KMeansClustering analysis with >>> cep streams with samoa ml topologies. >>> And also i could not find a maven repo for samoa 0.4.0 incubating. So i >>> am currently using my local m2 repo's samoa 0..4.0 incubating for my >>> dependencies to work. The local one is built by original samoa source.thank >>> you. >>> >>> regards, >>> Mahesh. >>> >>> [1] >>> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/samoa/streaming/src/main/java/org/gsoc/samoa/streaming >>> >>> >>> On Mon, Jul 18, 2016 at 8:32 AM, Maheshakya Wijewardena < >>> [email protected]> wrote: >>> >>>> Hi Mahesh, >>>> >>>> Can you please share your samoa project? >>>> >>>> On Sun, Jul 17, 2016 at 11:19 AM, Mahesh Dananjaya < >>>> [email protected]> wrote: >>>> >>>>> >>>>> ---------- Forwarded message ---------- >>>>> From: Mahesh Dananjaya <[email protected]> >>>>> Date: Sun, Jul 17, 2016 at 11:18 AM >>>>> Subject: Re: GSOC2016: [ML][CEP] [SAMOA]Predictive analytic with >>>>> online data for WSO2 Machine Learner-Samoa Integration >>>>> To: Maheshakya Wijewardena <[email protected]> >>>>> >>>>> >>>>> Hi Maheshakaya, >>>>> just need a little help. In Samoa when we want to run a class what is >>>>> does it used this commands [1], >>>>> 1. bin/samoa storm target/SAMOA-Storm-0.0.1-SNAPSHOT.jar >>>>> "ClusteringEvaluation" >>>>> 2. bin/samoa storm target/SAMOA-Storm-0.0.1-SNAPSHOT.jar >>>>> "PrequentialEvaluation -d /tmp/dump.csv -i 1000000 -f 100000 -l >>>>> (classifiers.trees.VerticalHoeffdingTree -p 4) -s >>>>> (generators.RandomTreeGenerator -c 2 -o 10 -u 10)" >>>>> >>>>> what is does is call a class named LocalDoTask [4] and pass this >>>>> string as argument.After that that LocalDoTask call the relevent Tasks >>>>> such >>>>> as ClusteringEvaluation or PrequentialEvaluation. [2]. >>>>> >>>>> Now i have add samoa dependencies to my new maven project, where i >>>>> used original samoa source to write examples and test then earlier.Now i >>>>> want to push them into my new java project with samoa dependencies. I >>>>> added >>>>> dependency and it was built fine. Now i am calling my local DoTask.java >>>>> [3] >>>>> file as same as i did with samoa with, >>>>> java -cp target/streaming-1.0-SNAPSHOT.jar >>>>> org.gsoc.samoa.streaming.DoTask >>>>> "org.gsoc.samoa.streaming.ClusteringEvaluation" >>>>> But seems to be i am incorrect in some place. >>>>> Error: A JNI error has occurred, please check your installation and >>>>> try again >>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>> org/apache/samoa/topology/ComponentFactory >>>>> at java.lang.Class.getDeclaredMethods0(Native Method) >>>>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) >>>>> at java.lang.Class.privateGetMethodRecursive(Class.java:3048) >>>>> at java.lang.Class.getMethod0(Class.java:3018) >>>>> at java.lang.Class.getMethod(Class.java:1784) >>>>> at >>>>> sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) >>>>> at >>>>> sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) >>>>> Caused by: java.lang.ClassNotFoundException: >>>>> org.apache.samoa.topology.ComponentFactory >>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>> ... 7 more >>>>> >>>>> >>>>> can i actually call the Task like this. >>>>> >>>>> BR, >>>>> Mahesh. >>>>> >>>>> [1] >>>>> https://samoa.incubator.apache.org/documentation/Prequential-Evaluation-Task.html >>>>> [2] >>>>> https://github.com/apache/incubator-samoa/blob/releases/0.4.0-incubating-RC0/samoa-api/src/main/java/org/apache/samoa/tasks/ClusteringEvaluation.java >>>>> [3] >>>>> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/samoa/streaming/src/main/java/org/gsoc/samoa/streaming >>>>> [4] >>>>> https://github.com/apache/incubator-samoa/tree/releases/0.4.0-incubating-RC0/samoa-local/src/main/java/org/apache/samoa >>>>> >>>>> >>>>> On Thu, Jul 14, 2016 at 3:47 PM, Mahesh Dananjaya < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi srinath, >>>>>> sure.i am working on it.thank you. >>>>>> regards, >>>>>> Mahesh. >>>>>> >>>>>> On Thu, Jul 14, 2016 at 11:12 AM, Srinath Perera <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Mahesh, >>>>>>> >>>>>>> Let's focus on getting SAOMA work with CEP. It is OK to be limited >>>>>>> to few algorithms. >>>>>>> >>>>>>> --Srinath >>>>>>> >>>>>>> On Thu, Jul 14, 2016 at 10:49 AM, Mahesh Dananjaya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Maheshakya, >>>>>>>> I think we can build new tasks [1] like the one in execution plan >>>>>>>> in cep with samoa. I will try to build a one. >>>>>>>> regards, >>>>>>>> Mahesh. >>>>>>>> [1] >>>>>>>> https://samoa.incubator.apache.org/documentation/Developing-New-Tasks-in-SAMOA.html >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jul 14, 2016 at 10:35 AM, Mahesh Dananjaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Maheshakya, >>>>>>>>> I am building and running samoa to see its functionality. In samoa >>>>>>>>> still we have limited supports in algorithms. Samoa supports only >>>>>>>>> classification and clustering with streams. It also use kind of >>>>>>>>> StreamProcessor, like the one we use in StreamProcessor extension. I >>>>>>>>> was >>>>>>>>> getting started with Samoa referring to this page [1]. Then i ran >>>>>>>>> couple of >>>>>>>>> examples to identified the flow. Samoa use hadoop framework instead >>>>>>>>> spark >>>>>>>>> for distribution. But i am using it in a local mode. When i see the >>>>>>>>> Samoa >>>>>>>>> core there is only limited algorithms. IMO if we are going to use >>>>>>>>> Samoa we >>>>>>>>> have to limit the functionality and algorithms [2]. When i go to >>>>>>>>> developer >>>>>>>>> corner in [3], it seems to be something like CEP extension that we are >>>>>>>>> using currenlty. SO in Samoa though the algorihtms are limited, they >>>>>>>>> have >>>>>>>>> implemented streaming support for them. Therefore if we integrate it >>>>>>>>> into >>>>>>>>> CEP we have to look for how to handle streams and algorithms in Samoa >>>>>>>>> side. >>>>>>>>> Is it good for your side to have both hadoop and spark running >>>>>>>>> background.thank you. >>>>>>>>> regards, >>>>>>>>> Mahesh. >>>>>>>>> >>>>>>>>> [1] https://samoa.incubator.apache.org/documentation/Home.html >>>>>>>>> [2] >>>>>>>>> https://samoa.incubator.apache.org/documentation/api/current/index.html >>>>>>>>> [3] >>>>>>>>> https://samoa.incubator.apache.org/documentation/SAMOA-Topology.html >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ============================ >>>>>>> Srinath Perera, Ph.D. >>>>>>> http://people.apache.org/~hemapani/ >>>>>>> http://srinathsview.blogspot.com/ >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Pruthuvi Maheshakya Wijewardena >>>> >>>> [email protected] >>>> +94711228855 >>>> >>>> >>>> >>> >> > > > -- > ============================ > Srinath Perera, Ph.D. > http://people.apache.org/~hemapani/ > http://srinathsview.blogspot.com/ >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
