Hi, I've updated the thread "Cross Tenant Data Reading from Spark Queries in DAS", where I've done the implementation on writing back to the tenant space also from the super tenant space as well. This basically just eliminates the current requirement of executing identical Spark scripts in all the tenants, which would not be desirable performance wise, and now, we can just use super tenant's single script to process all the tenant's data and write back the results to their space. Of course, we've to now change the current ESB analytics scripts to use this feature, but it shouldn't be a big change. @Gihan/Supun, can we please check on this and get it done.
Cheers, Anjana. On Tue, Jun 21, 2016 at 12:35 PM, Anjana Fernando <anj...@wso2.com> wrote: > Hi, > > Yeah, as Srinath mentioned, we do have MT support there, where each ESB > analytics data/queryeis/dashboards would be fully isolated. This is the > usual desirable mode, when different tenants deploy their own analytics > scripts and all, and there is no connection at all with other tenants, and > obviously there shouldn't be. But in our solutions like this, where we > would deploy the exact same solution in each tenant, we can potentially > optimize the scenario by, only doing the processing in one place. This is > actually how APIM batch queries are written at the moment. Where they > altogether stores all the tenant's data in super tenant space and process > and put the summarized there itself. This model works, but then, if the > tenant itself wants to do their own processing it doesn't have access to > the data, that was suppose to be in his space. So a solution for this is > done, which is explained in detail in the mail "Cross Tenant Data Reading > from Spark Queries in DAS", where optionally the super tenant can read > tenant's data which creates a single virtual table view which contains data > from all the tenants. From this, you can read the data, do the processing > and store it in a table in super tenant space. In this approach, you will > still have to have the dashboard in the super tenant space, since the data > resides in the super tenant space. I will be doing some experiments to see > on the feasibility of storing back summarized data in tenant space soon. > > Even though we can get the batch scenario working efficiently, there would > be a problem in the CEP side, which we now use heavily in the product > analytics scenario processing. At the moment, there is no concept of having > a single CEP execution plan for all the tenants. When data is published to > receivers, they will be dispatched to their own streams and execution > plans, so we will need to find an approach for this, to optimize the > processing of the CEP side of things. Or else, I'm not sure, since these > are in-memory processing constructs, if having copies of the same execution > plans will actually be okay, compared to having a single one, since the > traffic is anyway split between them. > > Cheers, > Anjana. > > On Tue, Jun 21, 2016 at 9:35 AM, Srinath Perera <hemap...@gmail.com> > wrote: > >> We had a chat about this. >> >> This is supported, however, current implementation will run spark jobs >> per each tenant. So it will have performance concerns with a large number >> of tenants. >> >> Anjana, can you give the details and potential solutions? DAS team has >> implemented the support for reading tenant data from super tenant so we can >> run a one Spark Job for all tenants. But we will need to change the queries >> and figure out how tenant dashboards can get access to results in super >> tenant's space. >> >> Can we do a 5.1 addressing this? >> >> --Srinath >> >> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri <ka...@wso2.com> wrote: >> >>> Please note that this will be an essential feature for our integration >>> cloud. So, we need to decide how should we proceed with this soon. >>> >>> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri <ka...@wso2.com> wrote: >>> >>>> How about the $subject? I don't think there's no any limitation from >>>> the mediation engine side on collecting these stats. We need to have this >>>> for ESB 5. >>>> >>>> -- >>>> Kasun Indrasiri >>>> Software Architect >>>> WSO2, Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> cell: +94 77 556 5206 >>>> Blog : http://kasunpanorama.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> Kasun Indrasiri >>> Software Architect >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> cell: +94 77 556 5206 >>> Blog : http://kasunpanorama.blogspot.com/ >>> >> >> >> >> -- >> ============================ >> WSO2 Inc. http://wso2.com >> Blog: http://srinathsview.blogspot.com/ >> Photos: http://www.flickr.com/photos/hemapani/ twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
