On Fri, Oct 24, 2014 at 3:13 PM, Lasantha Fernando <[email protected]> wrote:
> Hi Maninda, > > On 24 October 2014 14:40, Maninda Edirisooriya <[email protected]> wrote: > >> >> On Fri, Oct 24, 2014 at 1:10 PM, Lasantha Fernando <[email protected]> >> wrote: >> >>> Hi Maninda, all, >>> >>> On 24 October 2014 12:39, Gihan Anuruddha <[email protected]> wrote: >>> >>>> We need to maintain a single location to give publisher credential >>>> details. At the moment we are maintaining same details in a different >>>> location that is a bit inconvenient. As an example if we change BAM >>>> url then we have to change BAM server profiles as well as mediation stat >>>> publish UI as well. Also, this password should support secureVault. >>>> >>> >>> +1 >>> >> Yes. When we can come up with the generic artifact which describes data >> sink URL and credentials, we can access them for all ESB components (like >> new mediator and mediation stats publisher) via a declarative service. As >> this artifact is dep synced across all the products other products too can >> get URL and credentials without any issue. All you have to do is to install >> the feature to that product that exposes that declarative service. >> >>> >>> >>>> >>>> On Fri, Oct 24, 2014 at 12:25 PM, Maninda Edirisooriya < >>>> [email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Today morning we had a discussion where Anjana, Lahiru, Vijitha, Kasun >>>>> and I participated. >>>>> >>>>> There we decided to develop a common Data Publisher / Data Sink axis2 >>>>> artifact which is generic throughout the platform. This will enable any >>>>> other carbon products also for finding data publishing connection / >>>>> credential details when they want to publish data to CEP / BAM. >>>>> >>>> >>> Are we going to couple with Axis2 for the data publisher/ data sink? >>> Querying about this since you had mentioned 'axis2 artifacts'.. :-). >>> >> I am not sure about the new artifact type that is going to replace axis2 >> artifact deployment, and that is the reason I mentioned the name "axis2". >> Yes it is better if we can go with the new deployment artifact type. :) >> > > In the current data-bridge model, we are not having Axis2 deployment > artifacts since the data receivers/data publishers are not hot-deployable > artifacts. In the current model, usually, we configure a single data > receiver for a server before start up and the no. of publishers , type are > also pre-configured and not hot-deployable. Are we going for a model with > multiple data publisher/data receiver artifacts deployed through an Axis2 > deployer or similar? > You are correct. We can use a configuration file for configuring Data Publishing server connection URL and credentials assuming all the tenants are publishing to the same BAM / CEP server in a multi-tenant environment. > > >>> Can't we make it generic to the platform without coupling with Axis2 >>> since we are moving away from Axis2 in other areas of the platform? >>> >> Can you explain how to make it generic? This is really a problem I >> thought. I think we have to decide it before we completely get rid of >> Axis2. Maybe Azeez have an idea. >> >> Azeez, what do you think about the new artifact deployment modal >> independent from Axis2? >> >>> >>> >>>>> And specific to the ESB we are going to develop a mediator (it's name >>>>> is not confirmed yet) which will be similar to the current BAM mediator >>>>> which will refer to the above mentioned artifact for credentials and >>>>> connection parameters. As also we should include the transport we are >>>>> referring in that particular data sink / data publisher. For the first cut >>>>> we can use only the Thrift and hope to extend for HTTP and JMS in future. >>>>> >>>>> In new implementation of the mediator, there will be no configuration >>>>> for Stream configuration like the BAM mediator. Instead there will be >>>>> inline configurations in the mediator XML which consists of >>>>> 1. a stream ID (or the combination of Stream Name and Stream Version) >>>>> 2. and mappings from message XPath to the stream field. (i.e. maps >>>>> from message properties to the stream fields.) >>>>> >>>> >>> +1 for mapping message to stream attributes as described above. >>> >>> >>>>> For the convenience to the user, we can prompt the stream fields for >>>>> mapping from the UI, (once stream ID is provided) if the stream is already >>>>> there in the stream definition store. Anyway there (if want to get UI >>>>> help) >>>>> we have to use the existing streams that were already created from CEP or >>>>> BAM. But still you can add a mapping for a new stream which can >>>>> subsequently be added to the stream definition store from CEP or BAM. >>>>> >>>>> We have decided to go for a mediator instead of a connector as this >>>>> has nothing to do for accessing an external API kind of operation but to >>>>> publish data for our own products according to the connection information >>>>> given by the internal Data Publisher / Data Sink artifact. >>>>> >>>>> WDYT about the cases when we use other transports like HTTP, JMS, MQTT >>>>> etc. ? >>>>> >>>> >>> Will we really be needing HTTP, JMS support when communicating within >>> the platform across different products? Asking this since we already have >>> similar functionality in many places (e.g. ESB, BAM/CEP input adaptors) and >>> our general architectural approach is to get the events/messages into the >>> platform from one product, then communicate within the platform using a >>> single native format for efficiency/performance. WDYT? >>> >> >> Due to the client loadbalancing requirement of the Thrift endpoint there >> may be some limitations when it comes to publishing from one cluster to >> another cluster where data receiver thrift URLs are not visible outside >> from the second cluster. There we may have to use HTTP data sink type. And >> if there is a requirement for publishing to a 3rd party JMS topic we may >> need JMS support and so on. Anyway we should first focus on the Thrift type >> and will think about others as the next steps. :) >> > > If we are trying to send something over HTTP or JMS from ESB, can't we > make use of the already existing implementations within ESB to send to HTTP > or JMS endpoints? It will indeed be very useful to have a set of pluggable > transports that can be used by any product of the platform. I think the > concern is whether we will be redoing already available transports when > implementing HTTP/JMS for databridge level as well. Of course, I might have > misunderstood the approach mentioned here, if so, please correct me if am > wrong.. :-) > In this data sink artifact we will build the messages according to the events used in the stream where the existing HTTP, JMS and etc. message endpoints are capable of sending raw messages. The example is Entitlement mediator where it sends data via HTTP and with a specific format. > > Thanks, > Lasantha > > >> >>> Thanks, >>> Lasantha >>> >>>> >>>>> Feedback is welcome. >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> *Maninda Edirisooriya* >>>>> Senior Software Engineer >>>>> >>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>> >>>>> *Blog* : http://maninda.blogspot.com/ >>>>> *E-mail* : [email protected] >>>>> *Skype* : @manindae >>>>> *Twitter* : @maninda >>>>> >>>>> On Mon, Oct 20, 2014 at 8:00 PM, Sriskandarajah Suhothayan < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Oct 20, 2014 at 4:06 AM, Mohanadarshan Vivekanandalingam < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> We had a discussion sometime back (last year) on this.. Please check >>>>>>> the mail thread [1] for more info.. >>>>>>> >>>>>>> >>>>>> Yes, I believe we all agree that we need to improve the current BAM >>>>>> Mediator. >>>>>> When I had a chat with kasun on this he suggested writing it as a >>>>>> connector will be much cleaner. >>>>>> >>>>>> Where we can have the connection information and stream definition in >>>>>> <init> and <defineStream> then then pass the data via <publish>. >>>>>> >>>>>> I believe its better to have a f2f meeting with kasun on this. >>>>>> >>>>>> Regards >>>>>> Suho >>>>>> >>>>>> [1] [Architecture] Improving Data Publisher Mediator to replace BAM >>>>>>> Mediator >>>>>>> >>>>>>> Thanks, >>>>>>> Mohan >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 20, 2014 at 12:55 PM, Anjana Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Sanjiva, >>>>>>>> >>>>>>>> On Sat, Oct 18, 2014 at 9:44 PM, Sanjiva Weerawarana < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> On Fri, Oct 17, 2014 at 6:47 PM, Lahiru Chandima <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> We (Vijitha<[email protected]> and myself) are going to develop >>>>>>>>>> an ESB Connector which can publish events to BAM and CEP. >>>>>>>>>> >>>>>>>>> >>>>>>>>> We're currently using the words "ESB Connector" to mean something >>>>>>>>> that becomes a set of mediators to access (primarily) HTTP APIs. This >>>>>>>>> is >>>>>>>>> not that right?? If so lets not use the same words please. >>>>>>>>> >>>>>>>> >>>>>>>> Yeah, I guess, this is simply a mediator, just like what we have >>>>>>>> now. It was just Kasun has mentioned to Suho, to create this as an ESB >>>>>>>> Connector, where we haven't actually gone into what it means to create >>>>>>>> a >>>>>>>> connector. @Kasun, maybe you can shed some light on this, on what you >>>>>>>> meant >>>>>>>> earlier. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Currently, BAM mediator is used for sending events from ESB to >>>>>>>>>> BAM. BAM Mediator needs to be configured with server details and >>>>>>>>>> streams >>>>>>>>>> need to be defined prior to using them in sequences of ESB Synapse >>>>>>>>>> configuration. >>>>>>>>>> >>>>>>>>>> With new ESB connector, it will be possible to add all >>>>>>>>>> configurations needed to publish events to BAM or CEP in the ESB >>>>>>>>>> Synapse >>>>>>>>>> configuration itself. This brings the configuration to a single place >>>>>>>>>> unlike when using the BAM mediator. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Is this a real requirement? Do we expect different ESB configs >>>>>>>>> (running on the same server) to want to send data to different BAM/CEP >>>>>>>>> endpoints? I understand the multitenancy usecase - is it only for >>>>>>>>> that? >>>>>>>>> >>>>>>>> >>>>>>>> Actually, the idea here is, what we have now for defining the >>>>>>>> target for events, "BAM Profiles", to replace that. Basically in a BAM >>>>>>>> Profile, we put all the information such as the event receiver >>>>>>>> details, and >>>>>>>> the stream definition, and how the properties are mapped using >>>>>>>> properties >>>>>>>> from a sequence. We want to clean this up by, simply having something >>>>>>>> equivalent to a message store we have now, where we named is "data >>>>>>>> sink" >>>>>>>> for now, to be the target for events, and the actual stream mapping >>>>>>>> will be >>>>>>>> done by the mediator itself, in the sequence. And the motivation for >>>>>>>> having >>>>>>>> this as an artifact in the synapse configuration is, where from a place >>>>>>>> like DevStudio or any development environment, we can create these >>>>>>>> deployment artifacts can directly deploy them, without going to the >>>>>>>> admin >>>>>>>> console and adding them through the UI. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> *High level architecture - components.* >>>>>>>>>> >>>>>>>>>> 1) New Synapse construct will be introduced to configure details >>>>>>>>>> related to the data sink (ie. CEP or BAM). >>>>>>>>>> >>>>>>>>>> Eg. >>>>>>>>>> >>>>>>>>>> <dataSink name=”bam_thrift_server”> >>>>>>>>>> <url>tcp://localhost:7611<url> >>>>>>>>>> <username>admin</username> >>>>>>>>>> <password>admin</password> >>>>>>>>>> </dataSink> >>>>>>>>>> >>>>>>>>> >>>>>>>>> What do we call this now? The term "dataSink" doesn't work for me >>>>>>>>> but that could be just me. >>>>>>>>> >>>>>>>> >>>>>>>> Yeah, we had a meeting sometime back to discuss this, and that day >>>>>>>> also, we couldn't decide on the name :) .. basically something related >>>>>>>> to a >>>>>>>> dataSink, any suggestions? :) .. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> 2) An ESB connector will be developed which can be used to publish >>>>>>>>>> events to a data sink. Connector will use data-bridge API (Thrift) >>>>>>>>>> for >>>>>>>>>> publishing events (It will be possible to introduce new transports >>>>>>>>>> other >>>>>>>>>> than Thrift later). >>>>>>>>>> >>>>>>>>> >>>>>>>>> Once again, what are the operations? Why do we want to write a >>>>>>>>> connector instead of using / improving the current "BAM mediator"? >>>>>>>>> >>>>>>>> >>>>>>>> So yeah, it will basically be improving/re-writing the current >>>>>>>> mediator, with a name like "dataPublish" or something, without any >>>>>>>> "BAM" >>>>>>>> part, and with the stream property mapping part added there itself, >>>>>>>> rather >>>>>>>> than having it in another place. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Thrift stream related data (stream name, version and stream >>>>>>>>>> properties) will be configured in the ESB connector configuration (in >>>>>>>>>> Synapse config), which is more logical compared to configuring it >>>>>>>>>> separately as done in the BAM mediator. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Can you give a specific example to illustrate the difference >>>>>>>>> please? >>>>>>>>> >>>>>>>> >>>>>>>> So basically in the current BAM mediator, we do not give the direct >>>>>>>> mapping to the stream properties there, we simply populate some message >>>>>>>> context properties and call the mediator. And the mediator simply >>>>>>>> looks up >>>>>>>> the information provided by the BAM Profile to do the actual mapping, >>>>>>>> where >>>>>>>> in our profile, we have given like Xpath expressions to say, what are >>>>>>>> the >>>>>>>> values in a specific target stream event. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Anjana. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Sanjiva. >>>>>>>>> -- >>>>>>>>> Sanjiva Weerawarana, Ph.D. >>>>>>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>>>>>> email: [email protected]; office: (+1 650 745 4499 | +94 11 214 >>>>>>>>> 5345) x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 >>>>>>>>> 650 265 8311 >>>>>>>>> blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva >>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Anjana Fernando* >>>>>>>> Senior Technical Lead >>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *V. Mohanadarshan* >>>>>>> *Software Engineer,* >>>>>>> *Data Technologies Team,* >>>>>>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>>>>>> *lean.enterprise.middleware.* >>>>>>> >>>>>>> email: [email protected] >>>>>>> phone:(+94) 771117673 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *S. Suhothayan* >>>>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>>>> *WSO2 Inc. *http://wso2.com >>>>>> * <http://wso2.com/>* >>>>>> lean . enterprise . middleware >>>>>> >>>>>> >>>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >>>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | >>>>>> linked-in: >>>>>> http://lk.linkedin.com/in/suhothayan >>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> W.G. Gihan Anuruddha >>>> Senior Software Engineer | WSO2, Inc. >>>> M: +94772272595 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> *Lasantha Fernando* >>> Software Engineer - Data Technologies Team >>> WSO2 Inc. http://wso2.com >>> >>> email: [email protected] >>> mobile: (+94) 71 5247551 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Lasantha Fernando* > Software Engineer - Data Technologies Team > WSO2 Inc. http://wso2.com > > email: [email protected] > mobile: (+94) 71 5247551 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
