Great! So that's for Spark.. You mentioned CEP as a big issue as well. Any hope to fix in CEP too ??
------------------------------------------------------------------------------------- *Isabelle Mauny* VP, Product Strategy - WSO2, Inc. - http://wso2.com/ email: isabe...@wso2.com - mobile (Spain) : +34 616050684 - mobile (Sri Lanka) +94 (0)774777663 Landline: +1 (650) 745 4499 (USA) or +94 (11) 214 534 (Sri Lanka) Extension : 7302 On Wed, Jun 22, 2016 at 11:50 AM, Srinath Perera <hemap...@gmail.com> wrote: > Perfect!! > > On Wed, Jun 22, 2016 at 1:58 PM, Anjana Fernando <anj...@wso2.com> wrote: > >> Hi, >> >> I've updated the thread "Cross Tenant Data Reading from Spark Queries in >> DAS", where I've done the implementation on writing back to the tenant >> space also from the super tenant space as well. This basically just >> eliminates the current requirement of executing identical Spark scripts in >> all the tenants, which would not be desirable performance wise, and now, we >> can just use super tenant's single script to process all the tenant's data >> and write back the results to their space. Of course, we've to now change >> the current ESB analytics scripts to use this feature, but it shouldn't be >> a big change. @Gihan/Supun, can we please check on this and get it done. >> >> Cheers, >> Anjana. >> >> On Tue, Jun 21, 2016 at 12:35 PM, Anjana Fernando <anj...@wso2.com> >> wrote: >> >>> Hi, >>> >>> Yeah, as Srinath mentioned, we do have MT support there, where each ESB >>> analytics data/queryeis/dashboards would be fully isolated. This is the >>> usual desirable mode, when different tenants deploy their own analytics >>> scripts and all, and there is no connection at all with other tenants, and >>> obviously there shouldn't be. But in our solutions like this, where we >>> would deploy the exact same solution in each tenant, we can potentially >>> optimize the scenario by, only doing the processing in one place. This is >>> actually how APIM batch queries are written at the moment. Where they >>> altogether stores all the tenant's data in super tenant space and process >>> and put the summarized there itself. This model works, but then, if the >>> tenant itself wants to do their own processing it doesn't have access to >>> the data, that was suppose to be in his space. So a solution for this is >>> done, which is explained in detail in the mail "Cross Tenant Data Reading >>> from Spark Queries in DAS", where optionally the super tenant can read >>> tenant's data which creates a single virtual table view which contains data >>> from all the tenants. From this, you can read the data, do the processing >>> and store it in a table in super tenant space. In this approach, you will >>> still have to have the dashboard in the super tenant space, since the data >>> resides in the super tenant space. I will be doing some experiments to see >>> on the feasibility of storing back summarized data in tenant space soon. >>> >>> Even though we can get the batch scenario working efficiently, there >>> would be a problem in the CEP side, which we now use heavily in the product >>> analytics scenario processing. At the moment, there is no concept of having >>> a single CEP execution plan for all the tenants. When data is published to >>> receivers, they will be dispatched to their own streams and execution >>> plans, so we will need to find an approach for this, to optimize the >>> processing of the CEP side of things. Or else, I'm not sure, since these >>> are in-memory processing constructs, if having copies of the same execution >>> plans will actually be okay, compared to having a single one, since the >>> traffic is anyway split between them. >>> >>> Cheers, >>> Anjana. >>> >>> On Tue, Jun 21, 2016 at 9:35 AM, Srinath Perera <hemap...@gmail.com> >>> wrote: >>> >>>> We had a chat about this. >>>> >>>> This is supported, however, current implementation will run spark jobs >>>> per each tenant. So it will have performance concerns with a large number >>>> of tenants. >>>> >>>> Anjana, can you give the details and potential solutions? DAS team has >>>> implemented the support for reading tenant data from super tenant so we can >>>> run a one Spark Job for all tenants. But we will need to change the queries >>>> and figure out how tenant dashboards can get access to results in super >>>> tenant's space. >>>> >>>> Can we do a 5.1 addressing this? >>>> >>>> --Srinath >>>> >>>> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri <ka...@wso2.com> >>>> wrote: >>>> >>>>> Please note that this will be an essential feature for our integration >>>>> cloud. So, we need to decide how should we proceed with this soon. >>>>> >>>>> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri <ka...@wso2.com> >>>>> wrote: >>>>> >>>>>> How about the $subject? I don't think there's no any limitation from >>>>>> the mediation engine side on collecting these stats. We need to have this >>>>>> for ESB 5. >>>>>> >>>>>> -- >>>>>> Kasun Indrasiri >>>>>> Software Architect >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> cell: +94 77 556 5206 >>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Kasun Indrasiri >>>>> Software Architect >>>>> WSO2, Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> cell: +94 77 556 5206 >>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>> >>>> >>>> >>>> >>>> -- >>>> ============================ >>>> WSO2 Inc. http://wso2.com >>>> Blog: http://srinathsview.blogspot.com/ >>>> Photos: http://www.flickr.com/photos/hemapani/ >>>> twitter:@srinath_perera >>>> Site: http://people.apache.org/~hemapani/ >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Senior Technical Lead >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> *Anjana Fernando* >> Senior Technical Lead >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> > > > > -- > ============================ > WSO2 Inc. http://wso2.com > Blog: http://srinathsview.blogspot.com/ > Photos: http://www.flickr.com/photos/hemapani/ twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
