If one tenant engages a task which hogs a server, all other tenants tasks
would be affected and potentially the server can get unresponsive. The
tenant partitioning you suggest can still cause this issue, right?
On Tue, May 22, 2012 at 5:02 PM, Anjana Fernando <anj...@wso2.com> wrote:
> Hi,
>
> After having a chat with Dimuthu, she suggested that we keep a flag in
> super tenant's registry mentioning if a specific tenant is having any
> tasks. So we can check this at the startup and only load the registries
> which has any. So there won't be any overhead of loading all the tenant
> registries. So I will go ahead with this approach and implement it. And
> also, shall I do the tenant partitioning approach I mentioned earlier? ..
>
> Cheers,
> Anjana.
>
>
> On Tue, May 22, 2012 at 3:40 PM, Anjana Fernando <anj...@wso2.com> wrote:
>
>> Hi Dimuthu,
>>
>> So, you were actually referring to the part where we list the tasks for a
>> specific tenant. Not only the part of retrieving the tenant id list.
>> TaskManager is there to listen for any new tasks getting created in the
>> cluster, so it should be their up and running from the beginning .. also
>> TaskManager is the one who checks if a specific tenant has any tasks to
>> schedule at first. That is, to check if a specific tenant has tasks, we
>> need to list out the tenant ids and we have to contact its registry to find
>> if there are any tasks! (the task definitions are saved in the registry) ..
>> So this operation is inevitable .. and that is why I proposed a tenant
>> partitioning approach in the previous reply.
>>
>> Cheers,
>> Anjana.
>>
>>
>> On Tue, May 22, 2012 at 3:28 PM, Dimuthu Leelarathne
>> <dimut...@wso2.com>wrote:
>>
>>> Hi Anjana,
>>>
>>> Please see the stack trace below. The issue here is we are doing
>>> something against the framework. When you are initializing all TaskManagers
>>> for all tenants at startup it calls the tenant governance registry of the
>>> tenant and loads all the realms in a loop. From UM side there is a DB call
>>> going at getTenantUserRealm for each call because it is not initialized
>>> before. This accounts for 2001 db calls from UM side only if we have 2000
>>> tenants.
>>>
>>> It could be even more if we account the registry initialization. What I
>>> am saying is, can we only initialize the TaskManagers for only the tenants
>>> with Tasks or a better design?
>>>
>>>
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at
>>> org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:184)
>>> at
>>> org.wso2.carbon.user.core.common.DefaultRealm.initializeObjects(DefaultRealm.java:147)
>>> at
>>> org.wso2.carbon.user.core.common.DefaultRealm.init(DefaultRealm.java:113)
>>> at
>>> org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:252)
>>> at
>>> org.wso2.carbon.user.core.common.DefaultRealmService.getTenantUserRealm(DefaultRealmService.java:183)
>>> at
>>> org.wso2.carbon.registry.core.session.UserRegistry.init(UserRegistry.java:262)
>>> at
>>> org.wso2.carbon.registry.core.session.UserRegistry.<init>(UserRegistry.java:222)
>>> at
>>> org.wso2.carbon.registry.core.session.UserRegistry.<init>(UserRegistry.java:201)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getUserRegistry(EmbeddedRegistryService.java:426)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getSystemRegistry(EmbeddedRegistryService.java:291)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getGovernanceSystemRegistry(EmbeddedRegistryService.java:516)
>>> at
>>> org.wso2.carbon.ntask.core.TaskUtils.getGovRegistryForTenant(TaskUtils.java:37)
>>> at
>>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.getTaskTypesForTenant(TaskServiceImpl.java:104)
>>> at
>>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.initTaskManagersForTenant(TaskServiceImpl.java:85)
>>> at
>>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.initTaskManagers(TaskServiceImpl.java:77)
>>> at
>>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.<init>(TaskServiceImpl.java:63)
>>> at
>>> org.wso2.carbon.ntask.core.internal.TasksDSComponent.activate(TasksDSComponent.java:68)
>>>
>>> thanks,
>>> dimuthu
>>>
>>>
>>>
>>> On Tue, May 22, 2012 at 3:20 PM, Anjana Fernando <anj...@wso2.com>wrote:
>>>
>>>> Hi Dimuthu,
>>>>
>>>> On Tue, May 22, 2012 at 3:12 PM, Dimuthu Leelarathne <dimut...@wso2.com
>>>> > wrote:
>>>>
>>>>> Hi Anjana,
>>>>>
>>>>>
>>>>> Say there are 2000 tenants.
>>>>>
>>>>> This is just one call, but after that you load all 2000 user realms in
>>>>> a iterator for each tenant. Meaning another 2000 db calls to check for
>>>>> custom realms. Now it is 2001 db calls.
>>>>>
>>>>
>>>> I'm sorry, I didn't understand what "custom realms" are. So the code I
>>>> use is like the following,
>>>>
>>>> Tenant[] tenants =
>>>> TasksDSComponent.getRealmService().getTenantManager().
>>>> getAllTenants();
>>>> for (Tenant tenant : tenants) {
>>>> tids.add(tenant.getId());
>>>> }
>>>>
>>>> So you mean, when I do "tenant.getId()" also, there is another database
>>>> call going out? ..
>>>>
>>>> Cheers,
>>>> Anjana.
>>>>
>>>>
>>>>> thanks,
>>>>> dimuthu
>>>>>
>>>>>
>>>>>
>>>>>> Cheers,
>>>>>> Anjana.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 22, 2012 at 2:59 PM, Anjana Fernando <anj...@wso2.com>wrote:
>>>>>>>
>>>>>>>> Hi Azeez,
>>>>>>>>
>>>>>>>> On Tue, May 22, 2012 at 2:27 PM, Afkham Azeez <az...@wso2.com>wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 22, 2012 at 2:22 PM, Anjana Fernando
>>>>>>>>> <anj...@wso2.com>wrote:
>>>>>>>>>
>>>>>>>>>> Hi Dimuthu,
>>>>>>>>>>
>>>>>>>>>> On Tue, May 22, 2012 at 2:15 PM, Dimuthu Leelarathne <
>>>>>>>>>> dimut...@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Currently
>>>>>>>>>>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl is
>>>>>>>>>>> initializing
>>>>>>>>>>> task managers for all tenants at startup. This is against our
>>>>>>>>>>> tenant lazy
>>>>>>>>>>> loading concepts.
>>>>>>>>>>>
>>>>>>>>>>> Please move tenant ntask initialization to a
>>>>>>>>>>> Axis2ConfigurationContextObserver. This will be called when
>>>>>>>>>>> TenanConfiguraitonContext is loaded.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> As I remember, this was already brought up by Azeez earlier. The
>>>>>>>>>> rationale is, a scheduled task is not a service. It is not something
>>>>>>>>>> that
>>>>>>>>>> is activated on-demand by a user in a specific tenant. But is
>>>>>>>>>> suppose to
>>>>>>>>>> work in the scheduled manner continuously. So when a user in a tenant
>>>>>>>>>> schedules a task, it should always run, and should not unload when
>>>>>>>>>> the
>>>>>>>>>> tenant is unloaded. And when starting the server also, all the tasks
>>>>>>>>>> should
>>>>>>>>>> continue.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But now we have a major problem if we have say 100,001 tenants. We
>>>>>>>>> will create this for all tenants unnecessarily. With the next
>>>>>>>>> release, we
>>>>>>>>> have tenant partitions defined at the LB level, which relies on the
>>>>>>>>> worker
>>>>>>>>> nodes to lazy load the tenants. There is no tenant partitioning
>>>>>>>>> concept at
>>>>>>>>> the worker node level. So, even though cluster as-0-100 loads tenants
>>>>>>>>> 0-100, the task thing will load all 100,001! We need to find a
>>>>>>>>> solution to
>>>>>>>>> this issue.
>>>>>>>>>
>>>>>>>>
>>>>>>>> OK, so this has actually turned into a tenant partitioning problem.
>>>>>>>> First of all, when you say initializing a TaskManager, it just means,
>>>>>>>> creating an object to represent a tasks for a specific tenant and a
>>>>>>>> connection to a centrally managed Group object in ZooKeeper. If the
>>>>>>>> tasks
>>>>>>>> are not scheduled, it will simply take a block of memory and just sit
>>>>>>>> there. And it will not load the tenants, I'm simply retrieving the
>>>>>>>> total
>>>>>>>> tenant id list and only loading the tenant aware task managers, it
>>>>>>>> should
>>>>>>>> not load other services and all. And also, the actual tasks will not be
>>>>>>>> scheduled in a single server, they will be distributed across the whole
>>>>>>>> cluster.
>>>>>>>>
>>>>>>>> So anyway, to support the tenant partitioning problem. I suggest
>>>>>>>> the following. Lets say we have several clusters to represent different
>>>>>>>> tenant partitions. For example, 3 clusters, to divide the tenants
>>>>>>>> equally.
>>>>>>>> When starting the server, we can set a Java system property saying, the
>>>>>>>> number of clusters used for tenant partitioning and the partitioning
>>>>>>>> cluster the current server will belong to. So for example
>>>>>>>> -Dtenant.partition.count=3 -Dmy.tenant.partition=2 (suggest suitable
>>>>>>>> names). So with these information at hand, when starting the server,
>>>>>>>> we can
>>>>>>>> start only the tasks suppose to be scheduled in this server (take
>>>>>>>> tenant_id
>>>>>>>> % tenant.partition.count and match it to my.tenant.partition), and also
>>>>>>>> when a new tenant is joining, we can do the same. Hopefully, this type
>>>>>>>> of a
>>>>>>>> method can be used for other type of similar services too. I can
>>>>>>>> implement
>>>>>>>> this in ntask now, if it is an acceptable solution.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Anjana.
>>>>>>>>
>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Anjana.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> thanks,
>>>>>>>>>>> dimuthu
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Dimuthu Leelarathne
>>>>>>>>>>> Technical Lead
>>>>>>>>>>>
>>>>>>>>>>> WSO2, Inc. (http://wso2.com)
>>>>>>>>>>> email: dimut...@wso2.com
>>>>>>>>>>>
>>>>>>>>>>> Lean . Enterprise . Middleware
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Anjana Fernando*
>>>>>>>>>> Associate Technical Lead
>>>>>>>>>> WSO2 Inc. | http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Dev mailing list
>>>>>>>>>> Dev@wso2.org
>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Afkham Azeez*
>>>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>>>>> * <http://www.apache.org/>**
>>>>>>>>> email: **az...@wso2.com* <az...@wso2.com>* cell: +94 77 3320919
>>>>>>>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>*
>>>>>>>>> twitter:
>>>>>>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
>>>>>>>>> *
>>>>>>>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez*
>>>>>>>>>
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> *Lean . Enterprise . Middleware*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Anjana Fernando*
>>>>>>>> Associate Technical Lead
>>>>>>>> WSO2 Inc. | http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Afkham Azeez*
>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>>> * <http://www.apache.org/>**
>>>>>>> email: **az...@wso2.com* <az...@wso2.com>* cell: +94 77 3320919
>>>>>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>*
>>>>>>> twitter:
>>>>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
>>>>>>> *
>>>>>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez*
>>>>>>> *
>>>>>>> *
>>>>>>> *Lean . Enterprise . Middleware*
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Anjana Fernando*
>>>>>> Associate Technical Lead
>>>>>> WSO2 Inc. | http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dimuthu Leelarathne
>>>>> Technical Lead
>>>>>
>>>>> WSO2, Inc. (http://wso2.com)
>>>>> email: dimut...@wso2.com
>>>>>
>>>>> Lean . Enterprise . Middleware
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Anjana Fernando*
>>>> Associate Technical Lead
>>>> WSO2 Inc. | http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>
>>>
>>>
>>> --
>>> Dimuthu Leelarathne
>>> Technical Lead
>>>
>>> WSO2, Inc. (http://wso2.com)
>>> email: dimut...@wso2.com
>>>
>>> Lean . Enterprise . Middleware
>>>
>>>
>>
>>
>> --
>> *Anjana Fernando*
>> Associate Technical Lead
>> WSO2 Inc. | http://wso2.com
>> lean . enterprise . middleware
>>
>
>
>
> --
> *Anjana Fernando*
> Associate Technical Lead
> WSO2 Inc. | http://wso2.com
> lean . enterprise . middleware
>
> _______________________________________________
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
--
Regards,
Tharindu
blog: http://mackiemathew.com/
M: +94777759908
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
