[
https://issues.apache.org/jira/browse/S4-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238711#comment-13238711
]
Matthieu Morel commented on S4-22:
----------------------------------
I have taken a new approach for this problem, by porting the twitter example
from S4 0.3, and using an adapter cluster and a processing cluster.
I also created several scripts for launching zookeeper, uploading cluster
configurations to it, packaging apps and deploying them. See "s4" command for
that. (it relies on other scripts that are generated through gradle)
See branch S4-22-rebased and readme files in twitter-adapter and
twitter-counter projects in test-apps. Also note that tools only seem to work
with gradle 1.0M6
The adapter-to-app communication uses the concept of a "remote" stream, which
actually sends events to the specified app cluster.
2 conclusions:
1/ we need to have pluggable partitioning schemes (in order to perform some
balancing from the adapter to the app cluster for instance)
2/ I found an important issue with the multiclassloader approach. Indeed, since
events must be deserialized by the comm layer, the comm layer needs to be aware
of the event classes. Which is not the case if we load application classes at
the application level only. I'm not sure what path to take: removing the
multiclassloader may lead to conflicts between packaged apps and the platform...
Currently the twitter example uses a workaround: only generic events, but this
is a strong limitation.
> Adaptor
> -------
>
> Key: S4-22
> URL: https://issues.apache.org/jira/browse/S4-22
> Project: Apache S4
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Leo Neumeyer
> Assignee: Bruce Robbins
> Fix For: 0.5
>
> Attachments: s4-subclusters.pdf
>
>
> Need an adaptor for v0.5
> Idea I posted earlier:
> What do you think of this idea for a simple adaptor:
> - Adaptor extends App
> - Adaptor can send events but not receive (for now)
> - Adaptor is deployed as a regular App to the S4 cluster and as an
> Adaptor type in a host (separate from the S4 cluster).
> - Adaptor, unlike regular apps, can accept event data (in any format)
> directly, not via comm layer.
> - Input data is transformed into S4 events using a modular approach
> and by providing standard modules such as JSON.
> - Output events are exposed using EventSource and consumed by other
> apps without even knowing that they are Adaptors (only the App type is
> exposed in the cluster).
> - S4 events can be processed locally using PEs and Streams as usual.
> (We kind of need to get a local Sender for the local PEs and a
> standard cluster Sender for the EventSource object.)
> So why this approach?
> The GOOD:
> - Seems to be the least disruptive way to inject external events
> - Apps can easily consume the events in a modular way without any
> dependencies. Getting events from an adaptor or from another app is
> identical.
> - The adaptor would be packaged and deployed to the cluster as if it
> was an App (no incremental cost)
> - The adaptor can do preprocessing using the same programming model
> and can reuse PEs.
> The CHALLENGE:
> - We need to also deploy the Adaptor in a separate host. On the other
> hand, this is inevitable. At least we use the same approach instead of
> creating a different system.
> - The Adaptor will need to be integrated with ZK to get the physical
> addresses.
> - We need to deal with two senders.
> for later: two-way communication and adapter clusters.
> thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
