HI Malintha/Nuwan, I have few queries and like to provide few clarifications on this..
Actually most of the points that you guys raised are discussed before and changed based on some experiences.. - Defining stream from client side : We had this in old datapublisher component and removed due to various complications.. As Srinath mentioned this was done after many rounds of discussions.. - Having single configuration file to add DAS endpoint : In past, we had two level of configs (one for adapter which deals with transport and one for event formatting which deals with stream). That means we can use one event adapter for multiple streams.. But, there were some complaints saying it is not user friendly and should have one config to publish events.. And also, event publisher is also another deployable artifact like other synapse files.. In APIM, we are configuring key manager url in multiple api configs (revoke api and etc.. ), event publisher is also similar to that.. But, I do understand that your points are valid and think we have two options.. - Have a global property in output-adapter-config.xml and use that in Thrift event publisher (if that property exists).. - Allow to keep the DAS endpoint as a registry resource and use that.. Thanks, Mohan On Wed, Aug 24, 2016 at 10:05 AM, Nuwan Dias <[email protected]> wrote: > Hi, > > I spoke with Srinath regarding this too. It would be ideal to have a > global config which can override the per-stream configs so that we won't > have to maintain the DAS configs in multiple places. In all cases we've > come across so far we've only seen people use a single DAS (or cluster) and > publish all streams to that DAS cluster. So at least as of now I don't see > much value in having per stream config files. > > It would also be good if we can omit the necessity of having to define the > streams on the client side (because they're already defined on the server). > I see two main problems of having to define them on the clients. > > 1. When we have multiple event generators (a gateway cluster), the streams > will have to be defined on all nodes. Do we use dep-sync to sync up the > configs on the clients? Even if we do, I think its better to not depend on > dep-sync for the new stuff we're developing because we're moving away from > dep-sync soon. > > 2. Problems in upgrading to newer product versions. When upgrading the > product version, if a schema change has occurred, currently we only have to > worry about changing the schema on the server side. With the above > limitation, we now have to worry about changing the clients too. > > Thanks, > NuwanD. > > On Tue, Aug 23, 2016 at 10:16 PM, Malintha Amarasinghe <[email protected] > > wrote: > >> Hi All, >> >> Currently APIM is using an internal APIM specific configuration resides >> in api-manager.xml which include DAS server URL, username, password etc. >> Those configuration are used to instantiate an org.wso2.carbon.databridge. >> agent.DataPublisher object, which is then used to publish events >> directly to streams in DAS. >> >> As it is an APIM specific configuration, it is not reusable by other >> non-APIM features. Also org.wso2.carbon.databridge.agent.DataPublisher is >> intended to be used in non-carbon environments, for carbon environments, >> the recommended way is to use the Event Stream OSGi Service. >> >> So we were trying to re-structure APIM data publishing code to use new >> Event Stream OSGi service explained in [1]. >> >> 1. Create all the streams defined in DAS in APIM. >> 2. Create an *event publisher per each stream *which takes data from the >> stream and publish to DAS. >> 3. In each event publisher we need to configure DAS specific >> configuration. >> >> Ex: >> <?xml version="1.0" encoding="UTF-8"?> >> <eventPublisher name="test" statistics="disable" trace="disable" xmlns=" >> http://wso2.org/carbon/eventpublisher";> >> <from streamName="streamTest" version="1.0.0"/> >> <mapping customMapping="disable" type="wso2event"/> >> <to eventAdapterType="wso2event"> >> <property name="username">admin</property> >> <property name="protocol">thrift</property> >> <property name="publishingMode">non-blocking</property> >> <property name="publishTimeout">0</property> >> <property name="receiverURL">tcp://localhost:7611</property> >> <property encrypted="true" name="password">XXXXXXXX</property> >> </to> >> </eventPublisher> >> >> But we are facing a small problem here; in APIM there are many event >> streams being used. If there's a change happen to DAS configuration, we >> need to change it in all the event publishers which makes it difficult to >> maintain. As per the offline chat with DAS team, this is a current >> limitation. >> >> Are we going to move forward with the existing implementation? >> >> Thanks, >> Malintha >> >> [1] [Dev] Common configuration for publishing events from carbon servers >> to DAS/CEP >> -- >> Malintha Amarasinghe >> Software Engineer >> *WSO2, Inc. - lean | enterprise | middleware* >> http://wso2.com/ >> >> Mobile : +94 712383306 >> > > > > -- > Nuwan Dias > > Software Architect - WSO2, Inc. http://wso2.com > email : [email protected] > Phone : +94 777 775 729 > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *V. Mohanadarshan* *Associate Tech Lead,* *Data Technologies Team,* *WSO2, Inc. http://wso2.com <http://wso2.com> * *lean.enterprise.middleware.* email: [email protected] phone:(+94) 771117673
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
