[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608280#comment-15608280 ] ASF GitHub Bot commented on SAMOA-59: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-samoa/pull/54 @nicolas-kourtellis Thanks for the detailed review. I'll look into each item and get back to you. > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608004#comment-15608004 ] ASF GitHub Bot commented on SAMOA-59: - Github user nicolas-kourtellis commented on the issue: https://github.com/apache/incubator-samoa/pull/54 Hi @manuzhang, I managed to get the adapter working. Here are some notes that I would ask you take into consideration: - There are some inherent difficulties compiling gearpump from source. It would be good to have a compiled version to use directly. - Assuming this is given (which was my case because @manuzhang provided a compiled version), I managed to get samoa to compile/package with gearpump and run the package. - However, it would be good for the adapter to be upgraded to the new version of samoa in incubation, which is 0.5.0. But it should be fairly straightforward. This will allow us to test it with some more generators and ML methods added in the recent past. - Feedback when executing VHT: => The engine seems to continue executing the topology long after it has been created, used for the task and finished. Is there any way to pass a signal at the end of the execution to shut it down? (note: not the engine itself, but the topology). It was occupying resources on my computer for no reason at full CPU consumption. I found a manual way to kill it using the command "gear kill -appid X" with X being the id of the task, but I wonder if there is a more automatic way. => After I killed the jobs manually, the java processes that were created for the execution (I will assume they are the containers of the topologies) were still alive, just not consuming much resources. Shouldn't they have been terminated and removed? Is there a way to do that? => When I run new tasks, they just keep getting added on the engine (which is logical), even though I had killed the other ones earlier. => Multiple executions of the same experiment with the same seed for the random generator using the parameter -r which should yield the same random tree, perform differently with respect to accuracy. Is that expected? => Using a different seed for the random tree generator (r=1,...,5), the performance of the execution of VHT on local GearPump is fairly low (average over 5 different seeds: 65.39% accuracy) in comparison to running the topology on local Storm (84.046% accuracy). Any explanation why so much reduction in performance? > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574416#comment-15574416 ] ASF GitHub Bot commented on SAMOA-59: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-samoa/pull/54 @nicolas-kourtellis We haven't made a binary release yet so the binary for 0.8.1 is temporarily hosted at our old github repo https://github.com/gearpump/gearpump/releases/tag/0.8.1 > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569078#comment-15569078 ] ASF GitHub Bot commented on SAMOA-59: - Github user nicolas-kourtellis commented on the issue: https://github.com/apache/incubator-samoa/pull/54 Hi manuzhang, I was trying to test the adapter in local mode but realized that I am having difficulties compiling the gearpump 0.8.1 from source. I installed scala, sbt, etc., following the instructions I found here http://gearpump.incubator.apache.org/releases/latest/get-gearpump-distribution.html but it keeps complaining for some things and the build fails. Is it possible you make available the binary version of 0.8.1 so that I can use it directly? Or are you guys planning to make it available for the public soon? Thanks! Nicolas > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505343#comment-15505343 ] ASF GitHub Bot commented on SAMOA-59: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-samoa/pull/54 @nicolas-kourtellis any updates on this PR ? > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15484506#comment-15484506 ] ASF GitHub Bot commented on SAMOA-59: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-samoa/pull/54 @nicolas-kourtellis updated to latest release [0.8.1](https://dist.apache.org/repos/dist/release/incubator/gearpump/0.8.1-incubating/). Note this is a source release so you have to build gearpump from source. > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483727#comment-15483727 ] ASF GitHub Bot commented on SAMOA-59: - Github user nicolas-kourtellis commented on the issue: https://github.com/apache/incubator-samoa/pull/54 @manuzhang was there any update from your end on the version of gearpump that we can test Samoa? > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341272#comment-15341272 ] ASF GitHub Bot commented on SAMOA-59: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-samoa/pull/54 @gdfm I upgraded gearpump version to 0.8.1-SNAPSHOT which includes critical bug fixes and package renaming although that requires manual build. Hopefully we'll release gearpump 0.8.1 soon. > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318047#comment-15318047 ] ASF GitHub Bot commented on SAMOA-59: - Github user manuzhang commented on a diff in the pull request: https://github.com/apache/incubator-samoa/pull/54#discussion_r66021866 --- Diff: pom.xml --- @@ -127,6 +137,9 @@ 1.0.3 0.7.0 0.10.1 +0.7.5 --- End diff -- yes, I'll bump it to 0.8.0 > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SAMOA-59) Add an Adapter for Apache Gearpump
[ https://issues.apache.org/jira/browse/SAMOA-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240735#comment-15240735 ] ASF GitHub Bot commented on SAMOA-59: - GitHub user pangolulu opened a pull request: https://github.com/apache/incubator-samoa/pull/54 SAMOA-59, add an adapter for Apache Gearpump Hi, This PR is referenced to [SAMOA-59](https://issues.apache.org/jira/browse/SAMOA-59). And how to execute SAMOA with Apache Gearpump, you can follow the README file. Thanks. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pangolulu/incubator-samoa samoa-gearpump Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-samoa/pull/54.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #54 commit 1ff1f4044bf9f7e0ba129394af73f153bd1cb233 Author: pangolulu Date: 2016-04-07T11:45:27Z SAMOA-59, add an adapter for Apache Gearpump > Add an Adapter for Apache Gearpump > -- > > Key: SAMOA-59 > URL: https://issues.apache.org/jira/browse/SAMOA-59 > Project: SAMOA > Issue Type: New Feature >Reporter: Yu Gong > > Gearpump (http://www.gearpump.io/) is a real-time big data streaming engine. > It is inspired by recent advances in the Akka framework and a desire to > improve on existing streaming frameworks. Gearpump is event/message based and > featured as low latency handling, high performance, exactly once semantics, > dynamic topology update, Apache Storm compatibility, etc. It now becomes > Apache Incubator project > (http://incubator.apache.org/projects/gearpump.html). A Gearpump adapter for > SAMOA will translate the Apache SAMOA topologies into Gearpump DAGs in order > to run them on the Gearpump platform. So users can run streaming machine > learning algorithms built by Apache SAMOA on Gearpump. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
