On Fri, Oct 24, 2014 at 1:10 PM, Lasantha Fernando <[email protected]> wrote:
> Hi Maninda, all, > > On 24 October 2014 12:39, Gihan Anuruddha <[email protected]> wrote: > >> We need to maintain a single location to give publisher credential >> details. At the moment we are maintaining same details in a different >> location that is a bit inconvenient. As an example if we change BAM url >> then we have to change BAM server profiles as well as mediation stat >> publish UI as well. Also, this password should support secureVault. >> > > +1 > Yes. When we can come up with the generic artifact which describes data sink URL and credentials, we can access them for all ESB components (like new mediator and mediation stats publisher) via a declarative service. As this artifact is dep synced across all the products other products too can get URL and credentials without any issue. All you have to do is to install the feature to that product that exposes that declarative service. > > >> >> On Fri, Oct 24, 2014 at 12:25 PM, Maninda Edirisooriya <[email protected]> >> wrote: >> >>> Hi all, >>> >>> Today morning we had a discussion where Anjana, Lahiru, Vijitha, Kasun >>> and I participated. >>> >>> There we decided to develop a common Data Publisher / Data Sink axis2 >>> artifact which is generic throughout the platform. This will enable any >>> other carbon products also for finding data publishing connection / >>> credential details when they want to publish data to CEP / BAM. >>> >> > Are we going to couple with Axis2 for the data publisher/ data sink? > Querying about this since you had mentioned 'axis2 artifacts'.. :-). > I am not sure about the new artifact type that is going to replace axis2 artifact deployment, and that is the reason I mentioned the name "axis2". Yes it is better if we can go with the new deployment artifact type. :) > > Can't we make it generic to the platform without coupling with Axis2 since > we are moving away from Axis2 in other areas of the platform? > Can you explain how to make it generic? This is really a problem I thought. I think we have to decide it before we completely get rid of Axis2. Maybe Azeez have an idea. Azeez, what do you think about the new artifact deployment modal independent from Axis2? > > >>> And specific to the ESB we are going to develop a mediator (it's name is >>> not confirmed yet) which will be similar to the current BAM mediator which >>> will refer to the above mentioned artifact for credentials and connection >>> parameters. As also we should include the transport we are referring in >>> that particular data sink / data publisher. For the first cut we can use >>> only the Thrift and hope to extend for HTTP and JMS in future. >>> >>> In new implementation of the mediator, there will be no configuration >>> for Stream configuration like the BAM mediator. Instead there will be >>> inline configurations in the mediator XML which consists of >>> 1. a stream ID (or the combination of Stream Name and Stream Version) >>> 2. and mappings from message XPath to the stream field. (i.e. maps from >>> message properties to the stream fields.) >>> >> > +1 for mapping message to stream attributes as described above. > > >>> For the convenience to the user, we can prompt the stream fields for >>> mapping from the UI, (once stream ID is provided) if the stream is already >>> there in the stream definition store. Anyway there (if want to get UI help) >>> we have to use the existing streams that were already created from CEP or >>> BAM. But still you can add a mapping for a new stream which can >>> subsequently be added to the stream definition store from CEP or BAM. >>> >>> We have decided to go for a mediator instead of a connector as this has >>> nothing to do for accessing an external API kind of operation but to >>> publish data for our own products according to the connection information >>> given by the internal Data Publisher / Data Sink artifact. >>> >>> WDYT about the cases when we use other transports like HTTP, JMS, MQTT >>> etc. ? >>> >> > Will we really be needing HTTP, JMS support when communicating within the > platform across different products? Asking this since we already have > similar functionality in many places (e.g. ESB, BAM/CEP input adaptors) and > our general architectural approach is to get the events/messages into the > platform from one product, then communicate within the platform using a > single native format for efficiency/performance. WDYT? > Due to the client loadbalancing requirement of the Thrift endpoint there may be some limitations when it comes to publishing from one cluster to another cluster where data receiver thrift URLs are not visible outside from the second cluster. There we may have to use HTTP data sink type. And if there is a requirement for publishing to a 3rd party JMS topic we may need JMS support and so on. Anyway we should first focus on the Thrift type and will think about others as the next steps. :) > > Thanks, > Lasantha > >> >>> Feedback is welcome. >>> >>> Thanks. >>> >>> >>> *Maninda Edirisooriya* >>> Senior Software Engineer >>> >>> *WSO2, Inc.*lean.enterprise.middleware. >>> >>> *Blog* : http://maninda.blogspot.com/ >>> *E-mail* : [email protected] >>> *Skype* : @manindae >>> *Twitter* : @maninda >>> >>> On Mon, Oct 20, 2014 at 8:00 PM, Sriskandarajah Suhothayan < >>> [email protected]> wrote: >>> >>>> >>>> >>>> On Mon, Oct 20, 2014 at 4:06 AM, Mohanadarshan Vivekanandalingam < >>>> [email protected]> wrote: >>>> >>>>> Hi All, >>>>> >>>>> We had a discussion sometime back (last year) on this.. Please check >>>>> the mail thread [1] for more info.. >>>>> >>>>> >>>> Yes, I believe we all agree that we need to improve the current BAM >>>> Mediator. >>>> When I had a chat with kasun on this he suggested writing it as a >>>> connector will be much cleaner. >>>> >>>> Where we can have the connection information and stream definition in >>>> <init> and <defineStream> then then pass the data via <publish>. >>>> >>>> I believe its better to have a f2f meeting with kasun on this. >>>> >>>> Regards >>>> Suho >>>> >>>> [1] [Architecture] Improving Data Publisher Mediator to replace BAM >>>>> Mediator >>>>> >>>>> Thanks, >>>>> Mohan >>>>> >>>>> >>>>> On Mon, Oct 20, 2014 at 12:55 PM, Anjana Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Sanjiva, >>>>>> >>>>>> On Sat, Oct 18, 2014 at 9:44 PM, Sanjiva Weerawarana < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> On Fri, Oct 17, 2014 at 6:47 PM, Lahiru Chandima <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> We (Vijitha<[email protected]> and myself) are going to develop an >>>>>>>> ESB Connector which can publish events to BAM and CEP. >>>>>>>> >>>>>>> >>>>>>> We're currently using the words "ESB Connector" to mean something >>>>>>> that becomes a set of mediators to access (primarily) HTTP APIs. This is >>>>>>> not that right?? If so lets not use the same words please. >>>>>>> >>>>>> >>>>>> Yeah, I guess, this is simply a mediator, just like what we have now. >>>>>> It was just Kasun has mentioned to Suho, to create this as an ESB >>>>>> Connector, where we haven't actually gone into what it means to create a >>>>>> connector. @Kasun, maybe you can shed some light on this, on what you >>>>>> meant >>>>>> earlier. >>>>>> >>>>>> >>>>>>> >>>>>>> Currently, BAM mediator is used for sending events from ESB to BAM. >>>>>>>> BAM Mediator needs to be configured with server details and streams >>>>>>>> need to >>>>>>>> be defined prior to using them in sequences of ESB Synapse >>>>>>>> configuration. >>>>>>>> >>>>>>>> With new ESB connector, it will be possible to add all >>>>>>>> configurations needed to publish events to BAM or CEP in the ESB >>>>>>>> Synapse >>>>>>>> configuration itself. This brings the configuration to a single place >>>>>>>> unlike when using the BAM mediator. >>>>>>>> >>>>>>> >>>>>>> Is this a real requirement? Do we expect different ESB configs >>>>>>> (running on the same server) to want to send data to different BAM/CEP >>>>>>> endpoints? I understand the multitenancy usecase - is it only for that? >>>>>>> >>>>>> >>>>>> Actually, the idea here is, what we have now for defining the target >>>>>> for events, "BAM Profiles", to replace that. Basically in a BAM Profile, >>>>>> we >>>>>> put all the information such as the event receiver details, and the >>>>>> stream >>>>>> definition, and how the properties are mapped using properties from a >>>>>> sequence. We want to clean this up by, simply having something equivalent >>>>>> to a message store we have now, where we named is "data sink" for now, to >>>>>> be the target for events, and the actual stream mapping will be done by >>>>>> the >>>>>> mediator itself, in the sequence. And the motivation for having this as >>>>>> an >>>>>> artifact in the synapse configuration is, where from a place like >>>>>> DevStudio >>>>>> or any development environment, we can create these deployment artifacts >>>>>> can directly deploy them, without going to the admin console and adding >>>>>> them through the UI. >>>>>> >>>>>> >>>>>>> >>>>>>> *High level architecture - components.* >>>>>>>> >>>>>>>> 1) New Synapse construct will be introduced to configure details >>>>>>>> related to the data sink (ie. CEP or BAM). >>>>>>>> >>>>>>>> Eg. >>>>>>>> >>>>>>>> <dataSink name=”bam_thrift_server”> >>>>>>>> <url>tcp://localhost:7611<url> >>>>>>>> <username>admin</username> >>>>>>>> <password>admin</password> >>>>>>>> </dataSink> >>>>>>>> >>>>>>> >>>>>>> What do we call this now? The term "dataSink" doesn't work for me >>>>>>> but that could be just me. >>>>>>> >>>>>> >>>>>> Yeah, we had a meeting sometime back to discuss this, and that day >>>>>> also, we couldn't decide on the name :) .. basically something related >>>>>> to a >>>>>> dataSink, any suggestions? :) .. >>>>>> >>>>>> >>>>>>> >>>>>>> 2) An ESB connector will be developed which can be used to publish >>>>>>>> events to a data sink. Connector will use data-bridge API (Thrift) for >>>>>>>> publishing events (It will be possible to introduce new transports >>>>>>>> other >>>>>>>> than Thrift later). >>>>>>>> >>>>>>> >>>>>>> Once again, what are the operations? Why do we want to write a >>>>>>> connector instead of using / improving the current "BAM mediator"? >>>>>>> >>>>>> >>>>>> So yeah, it will basically be improving/re-writing the current >>>>>> mediator, with a name like "dataPublish" or something, without any "BAM" >>>>>> part, and with the stream property mapping part added there itself, >>>>>> rather >>>>>> than having it in another place. >>>>>> >>>>>> >>>>>>> >>>>>>> Thrift stream related data (stream name, version and stream >>>>>>>> properties) will be configured in the ESB connector configuration (in >>>>>>>> Synapse config), which is more logical compared to configuring it >>>>>>>> separately as done in the BAM mediator. >>>>>>>> >>>>>>> >>>>>>> Can you give a specific example to illustrate the difference please? >>>>>>> >>>>>> >>>>>> So basically in the current BAM mediator, we do not give the direct >>>>>> mapping to the stream properties there, we simply populate some message >>>>>> context properties and call the mediator. And the mediator simply looks >>>>>> up >>>>>> the information provided by the BAM Profile to do the actual mapping, >>>>>> where >>>>>> in our profile, we have given like Xpath expressions to say, what are the >>>>>> values in a specific target stream event. >>>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Sanjiva. >>>>>>> -- >>>>>>> Sanjiva Weerawarana, Ph.D. >>>>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>>>> email: [email protected]; office: (+1 650 745 4499 | +94 11 214 >>>>>>> 5345) x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 650 >>>>>>> 265 8311 >>>>>>> blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva >>>>>>> Lean . Enterprise . Middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Senior Technical Lead >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *V. Mohanadarshan* >>>>> *Software Engineer,* >>>>> *Data Technologies Team,* >>>>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>>>> *lean.enterprise.middleware.* >>>>> >>>>> email: [email protected] >>>>> phone:(+94) 771117673 >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> *S. Suhothayan* >>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>> *WSO2 Inc. *http://wso2.com >>>> * <http://wso2.com/>* >>>> lean . enterprise . middleware >>>> >>>> >>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >>>> http://lk.linkedin.com/in/suhothayan >>>> <http://lk.linkedin.com/in/suhothayan>* >>>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> W.G. Gihan Anuruddha >> Senior Software Engineer | WSO2, Inc. >> M: +94772272595 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Lasantha Fernando* > Software Engineer - Data Technologies Team > WSO2 Inc. http://wso2.com > > email: [email protected] > mobile: (+94) 71 5247551 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
