This plan looks good to me. We should also look beyond APIM 1.9.0 as well. I would personally like it if DAS had support for cross tenant scenarios as well. At least as a roadmap item, so we could use the DAS dashboards in the future versions of API Manager.
Thanks, NuwanD. On Wed, May 13, 2015 at 7:51 PM, Sinthuja Ragendran <[email protected]> wrote: > Hi all, > > We (anjana and me) had an internal discussion of going forward with APIM > 1.9 with DAS. Based on some recent discussions with sanjiva et all, there > were some changes to DAS analytics where it should be able to write into > other external RDBMS databases as default spark, other than DAL (Data > Access Layer of DAS), and this external data will be in human readable > format so that we can plug any dahboard/visualization layer. In our initial > plan, we only supported DAL to be datastore, and therefore APIM dashboard > will not work OOB as it will require to use some different APIs, and can't > use simply JDBC to connect to the datastore directly. But given that now > DAS will be able to insert data into other external databases in human > readable format, APIM dashboard will work OOB if the spark scripts are > written using the relevant spark connectors. > > Below is the summarization for complying APIM 1.9 with DAS 3.0. > > 1) APIM 1.9 will continue to push all tenant data into in super tenant > space, NO change required in the publishing method. > 2) During DAS 3.0 release we will make sure the other RDBMS connectors > supported by spark will work. > 3) Once DAS 3.0 is released, APIM scripts should be migrated to spark > script by using the relavant RDBMS connectos as mentioned in #2 and comply > with same summarized table format of APIM statistics. > 4) APIM 1.9 dashboard will be still used to visualize the summarized data, > and DAS dashboard cannot be used for APIM 1.9. > > Please raise if you have any concerns in above approach. > > Thanks, > Sinthuja. > > On Wed, May 13, 2015 at 9:50 AM, Nuwan Dias <[email protected]> wrote: > >> Hi all, >> >> So at least for now API Manager and APP Manager have use cases in which >> the summarized data span across tenants. An example scenario would be an >> API of tenant A being consumed by tenant B's application. In this case >> tenant B's dashboard needs to show his application data and tenant A's >> dashboard needs to show the API invocation data. The event is published by >> tenant A's API. >> >> If we were to use the DAS dashboard to achieve the above, what should be >> the recommended way of getting it done? i.e. What would be the best place >> to publish the data to and where should the analytics run? >> >> Thanks, >> NuwanD. >> >> On Wed, May 13, 2015 at 2:18 AM, Sinthuja Ragendran <[email protected]> >> wrote: >> >>> Hi sanjeewa, >>> >>> On Tue, May 12, 2015 at 7:39 PM, Sanjeewa Malalgoda <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On Tue, May 12, 2015 at 7:18 PM, Sinthuja Ragendran <[email protected]> >>>> wrote: >>>> >>>>> Hi sanjeewa, >>>>> >>>>> On Tue, May 12, 2015 at 7:02 PM, Sanjeewa Malalgoda <[email protected] >>>>> > wrote: >>>>> >>>>>> >>>>>> >>>>>> On Tue, May 12, 2015 at 6:07 PM, Sinthuja Ragendran < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Nuwan, >>>>>>> >>>>>>> On Tue, May 12, 2015 at 5:32 PM, Nuwan Dias <[email protected]> wrote: >>>>>>> >>>>>>>> Aren't we tightly coupling the data publisher and the analyzer by >>>>>>>> this? >>>>>>>> >>>>>>> >>>>>>> No we are not coupling data publishing with analytics here; By the >>>>>>> proposed method, it's simply you can send the tenanted data to >>>>>>> respective >>>>>>> tenant space, rather to single super tenant space. >>>>>>> >>>>>>> With the change you have proposed, a data published by a particular >>>>>>>> tenant would only be possible to analyze by the same tenant on DAS >>>>>>>> right? >>>>>>>> >>>>>>> >>>>>>> Yes, this is the same way currently other toolboxes behave. >>>>>>> Basically the tenant data will be only analyzed and visualized as per >>>>>>> tenant preference, therefore if the tenant foo.com prefers to >>>>>>> visualize the data then he/she should install the toolbox on DAS. >>>>>>> Ideally >>>>>>> IMHO there should be a way to enable to publishing/Monitoring per tenant >>>>>>> basis at APIM than enabling globally for all tenants. >>>>>>> >>>>>> I believe still we can write data publisher as we need and we can >>>>>> embed tenant ID to message payload. >>>>>> >>>>> >>>>> Yes, ofcourse. There is no lookup/validation on event payload or any >>>>> of the event attributes. You can send any type of events you want. >>>>> >>>>> Or is it something inherit from base class in a way data publisher >>>>>> implementation cannot override. >>>>>> >>>>> >>>>> The below is the example usage of the data publisher with proposed >>>>> change. >>>>> >>>>> DataPublisher dataPublisher = new DataPublisher(url, username, password); >>>>> dataPublisher.setTenantDomain("foo.com"); >>>>> dataPublisher.defineStream("streamDefn"); >>>>> dataPublisher.publish(new Event( streamId, System.currentTimeMillis(), >>>>> new Object[]{"external"}, null, >>>>> new Object[]{"test1", "test2"})); >>>>> >>>>> >>>>> Here a datapublisher object is specific to a tenant. Hence in multi >>>>> tenant case, you will have to create a separate data publishers for each >>>>> tenant and keep it in a map at APIM usege agent side, (Ex : Key - >>>>> tenantDomain, Value - DataPublisher) and then reuse the DataPublisher to >>>>> send the events for relevant tenants. Main advantage here is, you can send >>>>> to relevant tenant space, you don't need the tenant credentials rather >>>>> with >>>>> one valid privileged user credentials you can send it to respective >>>>> tenant's space.. >>>>> >>>>> Thanks for the clarification sinthuja. Yes it seems this is really >>>> cool feature. >>>> And if this works as you mentioned we can still publish events(all >>>> events belong to tenant) to super tenant space if need and have single >>>> toolbox(deployed in super tenant space) to summarize them. >>>> >>> >>> Yes. But DAS analytics and dashboard will be operating on tenant basis, >>> therefore if you send all events to super tenant, then only super tenant >>> can see from the dashboard if the summarized results go back to DAS's data >>> store. >>> >>> >>>> And if someone interested per tenant publishing and analyze we may do >>>> that as well. >>>> >>>> And small question, if we take tenant scenario let say tenant need to >>>> have their own data publisher. Is that something possible? Still they need >>>> admin credentials. >>>> >>> >>> Tenants always can use their own user name and password, and publish the >>> events which will finally go into their tenant space. They can't actually >>> use the above proposed model to switch between the tenants, because >>> switching between tenants is only permissible for super tenant who has a >>> valid permission, therefore they can't switch the tenant space and send it. >>> But of course, they can send to their own tenant space. >>> >>> Thanks, >>> Sinthuja. >>> >>> Thanks, >>> sanjeewa. >>> >>> >>> Thanks, >>>> Sinthuja >>>> >>>> If that is the case tenant ID will be picked automatically and send >>>>> along with payload. >>>>> >>>>> If we can set tenant ID in data publisher side then users can publish >>>>> event to other tenants space(event receiver may see same and push data >>>>> based on tenant ID present in payload). >>>>> How do we address that? >>>>> >>>> >>>>> Thanks, >>>>> sanjeewa. >>>>> >>>>>> >>>>>> >>>>>>> If that is the case it would cause us to live within a bunch of >>>>>>> restrictions which might hinder some of our use cases. See below for >>>>>>> some >>>>>>> scenarios. >>>>>>> >>>>>>> 1. The analyzer script will have to run on each tenant space, isn't >>>>>>> it? If so does someone need to manually deploy/configure it per each >>>>>>> tenant? >>>>>>> >>>>>> >>>>>> Deployment of toolbox is required on DAS. >>>>>> >>>>>>> >>>>>>> 2. In API Manager, we have use cases where a particular tenant's API >>>>>>> can be consumed by other tenant's applications. In this case, the stats >>>>>>> related to the API are shown under the API owning tenant's space. The >>>>>>> stats >>>>>>> related to the application are shown under the App owning tenant's >>>>>>> space. >>>>>>> But all stats are triggered from the API invocation. At the moment we >>>>>>> separate out the data based on the information available on the event >>>>>>> payload. So we can nicely summarize the data accordingly. >>>>>>> >>>>>> >>>>>> You still can send the respective tenant data as however you extract >>>>>> currently, but now it'll go to relevant tenant space, rather single super >>>>>> tenant space. >>>>>> >>>>>> >>>>>>> >>>>>>> If we do the determination of the tenant based on a value in the >>>>>>> event payload, would it not decouple the publisher and analyzer and >>>>>>> cater >>>>>>> for more possibilities in terms of data summarization/analytics? >>>>>>> >>>>>> >>>>>> Having the tenant id in the payload, and setting it in the data >>>>>> publisher doesn't have any difference in the analytics as far as both are >>>>>> going to be stored in respective tenant space. >>>>>> >>>>>> Here the only difference is in the publishing method, where you need >>>>>> to have separate publishers created for every tenant rather using the >>>>>> same >>>>>> publisher (you can have a map for tenantId, and datapublisher and then >>>>>> reuse the data publisher created for tenant). IMO since this is more >>>>>> cleaner approach, than deciding with convention of having specific >>>>>> 'TenantId' property name. But sending the tenantId in event payload is >>>>>> not >>>>>> going to have any difference in the analytics perspective, because once >>>>>> the >>>>>> relevant data is there fore relevant tenant space it's tenant's >>>>>> preference >>>>>> to deploy the toolbox and analyze it. >>>>>> >>>>>> Thanks, >>>>>> Sinthuja. >>>>>> >>>>>> Thanks, >>>>>> NuwanD. >>>>>> >>>>>>> >>>>>>> On Tue, May 12, 2015 at 11:38 PM, Sinthuja Ragendran < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> We (Isabelle, Anjana, Suho, NuwanD, Mohan, Iranga, Jaminda, and >>>>>>>> Myself) had a discussion about APIM 1.9 compatibility with DAS last >>>>>>>> week, >>>>>>>> and we need $subject to support APIM with DAS in tenant mode. As per >>>>>>>> APIM >>>>>>>> stats publishing to BAM 2.5.0, the events are published from super >>>>>>>> tenant >>>>>>>> (for all tenants) and having tenant id as separate field in the event. >>>>>>>> Therefore in BAM 2.5.0 the events are pushed in cassandra the >>>>>>>> super-tenant >>>>>>>> space, and toolbox and hive scripts runs on the super tenant space and >>>>>>>> having another field for tenant id in the summarized table as well. >>>>>>>> Finally >>>>>>>> dashboards in APIM will be connecting to one summarized table, and then >>>>>>>> filter results based on the tenant id for the respective tenants to >>>>>>>> visualize their statistics. >>>>>>>> >>>>>>>> DAS dashboard is going to live in DAS side rather APIM side, and >>>>>>>> hence the respective tenant users can only connect to their respective >>>>>>>> tenant data store. Therefore we need to have the statistics events to >>>>>>>> be >>>>>>>> pushed into respective tenant space, rather the super tenant space. >>>>>>>> >>>>>>>> To cater the above requirement, we will have to introduce below >>>>>>>> approach in DataPublisher. >>>>>>>> >>>>>>>> 1) Create data publisher with privileged super tenant user >>>>>>>> credentials. >>>>>>>> 2) setTenantId or setTenantDomain when you want to send the data on >>>>>>>> behalf of specific tenant. >>>>>>>> 3) publish events as usual. If the events are being received by BAM >>>>>>>> 2.5.0 then it will simply ignore the tenantId/domain part, and it will >>>>>>>> simply store the events in the logged in tenant space. In DAS the >>>>>>>> additional tenantId/domain will be considered, and it will be inserted >>>>>>>> in >>>>>>>> the relevant tenant space if the logged in user is a valid user with >>>>>>>> required permissions. >>>>>>>> >>>>>>>> For the above change to be used at APIM 1.9, we need to do a >>>>>>>> release for DataPublisher on 4.2.0 as APIM 1.9 is based on 4.2.0 >>>>>>>> carbon. >>>>>>>> I'll work on this, get the data publisher released soon. Let me know >>>>>>>> if you >>>>>>>> have any concerns in above approach. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sinthuja. >>>>>>>> >>>>>>>> -- >>>>>>>> *Sinthuja Rajendran* >>>>>>>> Associate Technical Lead >>>>>>>> WSO2, Inc.:http://wso2.com >>>>>>>> >>>>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>>>> Mobile: +94774273955 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Nuwan Dias >>>>>>> >>>>>>> Technical Lead - WSO2, Inc. http://wso2.com >>>>>>> email : [email protected] >>>>>>> Phone : +94 777 775 729 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Sinthuja Rajendran* >>>>>> Associate Technical Lead >>>>>> WSO2, Inc.:http://wso2.com >>>>>> >>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>> Mobile: +94774273955 >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *Sanjeewa Malalgoda* >>>>> WSO2 Inc. >>>>> Mobile : +94713068779 >>>>> >>>>> <http://sanjeewamalalgoda.blogspot.com/>blog >>>>> :http://sanjeewamalalgoda.blogspot.com/ >>>>> <http://sanjeewamalalgoda.blogspot.com/> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Sinthuja Rajendran* >>>> Associate Technical Lead >>>> WSO2, Inc.:http://wso2.com >>>> >>>> Blog: http://sinthu-rajan.blogspot.com/ >>>> Mobile: +94774273955 >>>> >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> *Sanjeewa Malalgoda* >>>> WSO2 Inc. >>>> Mobile : +94713068779 >>>> >>>> <http://sanjeewamalalgoda.blogspot.com/>blog >>>> :http://sanjeewamalalgoda.blogspot.com/ >>>> <http://sanjeewamalalgoda.blogspot.com/> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> *Sinthuja Rajendran* >>> Associate Technical Lead >>> WSO2, Inc.:http://wso2.com >>> >>> Blog: http://sinthu-rajan.blogspot.com/ >>> Mobile: +94774273955 >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Nuwan Dias >> >> Technical Lead - WSO2, Inc. http://wso2.com >> email : [email protected] >> Phone : +94 777 775 729 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Sinthuja Rajendran* > Associate Technical Lead > WSO2, Inc.:http://wso2.com > > Blog: http://sinthu-rajan.blogspot.com/ > Mobile: +94774273955 > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Nuwan Dias Technical Lead - WSO2, Inc. http://wso2.com email : [email protected] Phone : +94 777 775 729
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
