[
https://issues.apache.org/jira/browse/S4-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163735#comment-13163735
]
Leo Neumeyer commented on S4-22:
--------------------------------
I think that the issue is in the details, that is, what design leads to the
simplest API. I argue that for App2 to receive events from App1, we want App2
to have access to a local reference of App1. (Do we agree on this? otherwise I
may be missing something.) By deploying all apps in all logical nodes, we
accomplish that. I think this is an easy first step, we can later relax the
requirements. The only case in which this could be a problem is in a very large
cluster where it may be desirable not to update all the nodes. I doubt that
having gigantic clusters will be practical. Having smaller ones may be a better
option. In any case, we could relax the requirement as follows:
- Because App2 does not export its stream, it only needs to be in the cluster
in which it runs.
- Because App1 is only consumed by App2, App1 only needs to be deployed in the
cluster in which it runs and in the cluster in which App2 runs.
So symmetry is not an absolute requirement, we just need to determine the
dependencies and launch the app to the right nodes which is pretty neat.
However, I wouldn't worry about this right now. We can add more deployment
options later.
I think that the next step is to prototype this by writing App1 and App2 and
defining the APIs and process for resolving dependencies. I'll write something
soon.
-leo
> Adaptor
> -------
>
> Key: S4-22
> URL: https://issues.apache.org/jira/browse/S4-22
> Project: Apache S4
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Leo Neumeyer
> Assignee: Bruce Robbins
> Fix For: 0.5
>
> Attachments: s4-subclusters.pdf
>
>
> Need an adaptor for v0.5
> Idea I posted earlier:
> What do you think of this idea for a simple adaptor:
> - Adaptor extends App
> - Adaptor can send events but not receive (for now)
> - Adaptor is deployed as a regular App to the S4 cluster and as an
> Adaptor type in a host (separate from the S4 cluster).
> - Adaptor, unlike regular apps, can accept event data (in any format)
> directly, not via comm layer.
> - Input data is transformed into S4 events using a modular approach
> and by providing standard modules such as JSON.
> - Output events are exposed using EventSource and consumed by other
> apps without even knowing that they are Adaptors (only the App type is
> exposed in the cluster).
> - S4 events can be processed locally using PEs and Streams as usual.
> (We kind of need to get a local Sender for the local PEs and a
> standard cluster Sender for the EventSource object.)
> So why this approach?
> The GOOD:
> - Seems to be the least disruptive way to inject external events
> - Apps can easily consume the events in a modular way without any
> dependencies. Getting events from an adaptor or from another app is
> identical.
> - The adaptor would be packaged and deployed to the cluster as if it
> was an App (no incremental cost)
> - The adaptor can do preprocessing using the same programming model
> and can reuse PEs.
> The CHALLENGE:
> - We need to also deploy the Adaptor in a separate host. On the other
> hand, this is inevitable. At least we use the same approach instead of
> creating a different system.
> - The Adaptor will need to be integrated with ZK to get the physical
> addresses.
> - We need to deal with two senders.
> for later: two-way communication and adapter clusters.
> thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
