[jira] [Comment Edited] (FLINK-1421) Implement a SAMOA Adapter for Flink Streaming

[Fay Beligianni (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326196#comment-14326196
 ] 

Fay Beligianni edited comment on FLINK-1421 at 2/18/15 5:13 PM:


Hey Stefan,
Thank you very much for the response! Indeed that was the problem.\\
Regarding the Java serialization, yes it is not efficient but when we tried to 
use the Kryo serializer we were encountering problems with the tuples that we 
are streaming between the invokables.\\
Specifically, because we are streaming Tuple3String,UserDefinedObject,String 
elements, where UserDefinedClass is a custom implementation of the Samoa 
ContentEvent interface, we couldn't pass the TypeInformation of the custom 
object to the serializer, thus we had to use Java serialization.\\
For sure though we will try to come up with a more efficient solution for that 
issue.\\
P.S.: Thanks for the formatting hint!


was (Author: fobeligi):
Hey Stefan,
Thank you very much for the response! Indeed that was the problem.
Regarding the Java serialization, yes it is not efficient but when we tried to 
use the Kryo serializer we were encountering problems with the tuples that we 
are streaming between the invokables.
Specifically, because we are streaming Tuple3String,UserDefinedObject,String 
elements, where UserDefinedClass is a custom implementation of the Samoa 
ContentEvent interface, we couldn't pass the TypeInformation of the custom 
object to the serializer, thus we had to use Java serialization.
For sure though we will try to come up with a more efficient solution for that 
issue.
P.S.: Thanks for the formatting hint!

 Implement a SAMOA Adapter for Flink Streaming
 -

 Key: FLINK-1421
 URL: https://issues.apache.org/jira/browse/FLINK-1421
 Project: Flink
  Issue Type: New Feature
  Components: Streaming
Reporter: Paris Carbone
Assignee: Paris Carbone
   Original Estimate: 336h
  Remaining Estimate: 336h

 Yahoo's Samoa is an experimental incremental machine learning library that 
 builds on an abstract compositional data streaming model to write streaming 
 algorithms. The task is to provide an adapter from SAMOA topologies to 
 Flink-streaming job graphs in order to support Flink as a backend engine for 
 SAMOA tasks.
 A statup guide can be viewed here :
 https://docs.google.com/document/d/18glDJDYmnFGT1UGtZimaxZpGeeg1Ch14NgDoymhPk2A/pub
 The main working branch of the adapter :
 https://github.com/senorcarbone/samoa/tree/flink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-1421) Implement a SAMOA Adapter for Flink Streaming

[Paris Carbone (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289468#comment-14289468
 ] 

Paris Carbone edited comment on FLINK-1421 at 1/23/15 4:29 PM:
---

 :)
Samoa is a really interesting project, we can currently create flink 
datastreams from samoa topologies, just in cases where there are no cyclic 
dependencies. Apparently, cycles are quite common in Samoa tasks so we will 
support this very soon through Flink iterations. I bet the performance will 
also be comparatively good as well. I am really curious to run some 
cross-platform benchmarks as well.


was (Author: senorcarbone):
 :)
Samoa is a really interesting project, we can currently create flink 
datastreams from samoa topologies, just in cases where there are no cyclic 
dependencies. Apparently, cycles are quite common in Samoa tasks so we will 
support this very soon through Flink iterations.

 Implement a SAMOA Adapter for Flink Streaming
 -

 Key: FLINK-1421
 URL: https://issues.apache.org/jira/browse/FLINK-1421
 Project: Flink
  Issue Type: New Feature
  Components: Streaming
Reporter: Paris Carbone
Assignee: Paris Carbone
   Original Estimate: 336h
  Remaining Estimate: 336h

 Yahoo's Samoa is an experimental incremental machine learning library that 
 builds on an abstract compositional data streaming model to write streaming 
 algorithms. The task is to provide an adapter from SAMOA topologies to 
 Flink-streaming job graphs in order to support Flink as a backend engine for 
 SAMOA tasks.
 A statup guide can be viewed here :
 https://docs.google.com/document/d/18glDJDYmnFGT1UGtZimaxZpGeeg1Ch14NgDoymhPk2A/pub
 The main working branch of the adapter :
 https://github.com/senorcarbone/samoa/tree/flink



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)