Hi all,
In the current event builder architecture, it does both the conversion of
received events to a common format + mapping the input stream.
e.g.
- If the incoming stream is a databridge stream (say something like
userStatistics:1.0.0 { meta : [ipAddress, accessTime], correlation: [],
payload: [username, referrer, searchTerms, country] } ) and for processing
if we need only
filteredStats:1.0.0{meta:[ip],correlation:[],payload:[username, keywords]),
then the event builder will map accordingly and output the
filteredStats:1.0.0 stream.
- If we take something like incoming XML as an example, then there would
be an xpath like
//userStat:SearchEvent/userStat:ipAddress -> ip
//userStat:SearchEvent/userStat:username -> username
//userStat:SearchEvent/userStat:searchTerms -> keywords
In the CEP case, the mapping should be defined before writing the query. So
in the current architecture, the consumer of the event builder service will
subscribe only when the stream is needed. A notification will be sent to
any registered listeners when a new stream definition is available. One
event builder will always be exposing only one stream in the current
architecture (Multiple event builders can be configured to push events to
the same stream, but not the other way round).
AFAIU for some BAM usecases, I think that events coming through all streams
are needed. For certain other usecases, filtered/mapped streams are needed.
Since event builder component does not allow simply exposing all event
streams as they are using a single event builder instance, IMO the approach
of listening to both databridge and event-builder is a viable alternative.
However, is it only for data-bridge event streams that all streams are
needed, or will BAM need all streams from other transports (JMS, ws-event
etc.) as well? If so maybe, programmatically creating event builders might
be a more feasible approach. This approach would mean that each stream
would have an associated event builder. Am +1 for this if creating an event
builder per stream is not too much of an overhead. Currently, the
event-builder supports writing extensions. Maybe if a pass-through
event-builder is written with minimal overhead, would this approach be more
feasible?
Or maybe there is a cleaner way to deal with this other than these two
approaches?
Thanks,
Lasantha
On 14 August 2013 21:04, Malith Dhanushka <[email protected]> wrote:
>
>
>
> On Wed, Aug 14, 2013 at 1:07 PM, Maninda Edirisooriya <[email protected]>wrote:
>
>> At the moment in BAM, all the data received from data bridge are directly
>> stored to the Cassandra. When a new stream creation message is received,
>> that stream is automatically created and starts to listen to that stream
>> automatically.
>>
>> As a new feature we are going to re-use the InputTransportAdapter and
>> InputMessageBuilder (currently used by CEP) to receive messages to BAM. The
>> rationale is to unify the message receiving features in both CEP and BAM
>> which will enable BAM to receive message from other transports such as JMS.
>>
>> One problem with this new feature is enabling the automatic stream
>> creation with InputTransportAdapter and InputMessageBuilder. The user has
>> to pre-configure the streams it is intended to be listened so that new
>> message builders are created for each stream.
>>
>> The second problem is filtering the streams come to CEP and BAM. Some
>> streams may be only required to be used by CEP but not BAM. The reason is
>> keeping all the messages comes to CEP in Cassandra will waste the disk
>> space. But there is no stream filtering mechanism either in CEP or BAM at
>> the moment.
>>
>
> Yes. This is a limitation and there should be a proper mechanism to route
> streams based on stream content. This can be implemented as a generic
> feature.
>
>
>> The solution we have come up is as follows.
>>
>> 1. BAM is listening to both event builders and data bridge.
>> 2. Event builders should be pre configured if the streams have to be
>> received via event builders.
>> 3. But still all the streams come to data bridge will be received by the
>> data bridge listener.
>> 4. When a new stream creation message is received either to event builder
>> or to data bridge, they will notify it to our new lisetner.
>> 5. There is a separate configuration stating what are the streams
>> accepted by the BAM. If that configuration is "*", all the streams should
>> be accepted. (should go to Cassandra) If there is a comma seperated list of
>> streams, only those streams should go to Cassandra. If a set of streams
>> stated after a "!" symbol, all the streams except those streams should go
>> to Cassandra.
>> 6. When a stream creation message is notified to a listener, that
>> listener should create a stream in the Cassandra only if that stream is
>> matching with the stream set stated in the config file.
>> 7. Now the problem is receiving the same message from both event builder
>> and the data bridge. Therefore when a stream creation is notified, the
>> notifier (either event builder or the data bridge) is remembered by the
>> listener. Only that notifier will be allowed to store events to the
>> Cassandra using that stream. In other words, either event builder or the
>> data bridge can store events using a unique stream.
>>
>
> +1 for the approach.
>
>
>>
>>
>> *
>> Maninda Edirisooriya*
>> Software Engineer
>> *WSO2, Inc.
>> *lean.enterprise.middleware.
>>
>> *Blog* : http://maninda.blogspot.com/
>> *Phone* : +94 777603226
>>
>> _______________________________________________
>> Architecture mailing list
>> [email protected]
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Malith Dhanushka
>
> Engineer - Data Technologies
> *WSO2, Inc. : wso2.com*
>
> *Mobile* : +94 716 506 693
>
> _______________________________________________
> Architecture mailing list
> [email protected]
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>
--
*Lasantha Fernando*
Software Engineer - Data Technologies Team
WSO2 Inc. http://wso2.com
email: [email protected]
mobile: (+94) 71 5247551
_______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
