Hi All, First of all, I believe that we must accept the fact that there will be situations in which multiple tenants are hosted by the same OSGi container. I think there will be several use cases in which we cannot effort to start an isolated container for each tenant, for resource consumption reasons or other. That means that we will have to deal with the fact that some services will be tenant aware while others will not. If a developer creates a service that is tenant aware, a separate service instance will be available for each tenant within that container. For a tenant unaware service, there will be only one service instance running to be used by any tenant. Now the problem is how service dependencies between tenant aware and unaware services are resolved. If I created a tenant aware service and have a service dependency on another service, I need to know if that service is tenant aware as I need to define the 'tenantid' service property defining this dependency. But it might be the case that at the time I programmed my tenant aware service, this other service was not tenant aware but has been made tenant aware later on. So that means that we need to think of a way that even for a programmer, it is irrelevant if a service on which his service depends is tenant aware or not. He should just get the proper tenant aware service injected.
<braindump> Now I think the 'context' I'm currently working on might be the/a solution to this problem. In general a 'context' defines the context of an execution. In case of a HTTP request for example, it is a 'request context' and the current tenant could be resolved from the hostname and the current user from the token cookie. In case of a scheduled job, the context is a 'job execution context' and the job itself may define the tenant of that context. In case of a remote service, the context should be send along with the rpc. Anyway, I think that there is always some 'context' and that context determines, amongst many other things, the tenant that is 'active' in the current method invocation. Now if a service is tenant ware, this just means that it reads the current tenant from the available context and acts upon. This would add a little more complexity to the implementation of a tenant aware service as upon each method invocation it needs to pull this tenant information from the context, but we could simplify that for example by using AOP (the tenant is an aspect). However, if the context is available as a service (as I'm planning to), retrieving the current tenant from the context is rather easy and might do for the first implementation. </braindump> Please let me know what you think of this brain dump and if it could contribute to solving our 'tenant problem'. Regards, Ivo -----Original Message----- From: amdatu-developers-bounces at amdatu.org [mailto:[email protected]] On Behalf Of Marcel Offermans Sent: vrijdag 31 december 2010 11:18 To: amdatu-developers at amdatu.org Subject: Re: [Amdatu-developers] Multi-tenancy (and more) design Hello Mark, all, On 31 Dec 2010, at 10:04 , Mark Machielsen wrote: > In the last conference call we concluded that some services will be tenant > aware. For example the Cassandra bundle won't we deployed in every tenant > container because this would not scale. So we now have two options: a tenant > aware bundle, and a bundle which is not tenant aware and runs in every tenant > container. Agreed, in even more generic terms you could say that there are components that provide services (if you need more than one, you need to deploy multiple components) and components that provide service factories (which is our tenant aware option). Then there are different deployment options in the sense that you can deploy all components in the same container, or in different ones. We concluded that if a component is not tenant aware it needs to be deployed in a separate container per tenant. If it is, it can be deployed in one container and, via remote services, be published in each tenant specific container. > Question which arise: > - when to use which option When the savings in resource consumption are worth it, create a multi tenant aware component. In theory, we never need them, if each tenant gets its own container. However, we might not be using the available resources (memory, threads, ports, ...) effectively if we do it like that. At the container level that's why we're considering running multiple JVMs and multiple containers per JVM. At the component level, multi tenancy can help us pool resources. The downsides? Added complexity and individual service invocations will be slower. That last point is important, as normal service calls will be direct method invocations and these will be remote service calls. Even with an efficient implementation, those calls will be significantly slower so any gains of going for a multi tenant aware implementation should be big enough to counteract that. > - what are we going to communicate: is the default tenant aware or not The default, and easiest, is to not make your component tentant aware. > - is it explainable that there are two ways of developing I think so. And if you can't explain it to someone, just tell them there's only one way. ;) > - we should prevent that a bundle is developed as tenant unaware and must be > transformed to tenant aware because of scaling issues. Or worse: a tenant > unaware bundle can be used in a certain application, but not in another > application, because of a different deployment So when in general do you resolve performance issues? A lot of the time they are hard to predict. Of course that does not mean you cannot properly design a component, use scalable algorithms, etc. I don't think there is a generic recipe though. > - when we are building an application where all tenants are the same (for > example BC), why should we develop tenant unaware bundles (besides the > programming model) See above (downsides). However, if you deploy all tenant aware bundles in the same container and there effectively is no "container per tenant" then one of the downsides (slower invocations) can be avoided. If the added complexity is no issue, and the lack of isolation is not either, I see no reason not to do it like that. As a side note, if you're running an application for tenants A, B and C and suddenly A starts consuming a lot of resources, a cluster manager might decide to migrate A to a different container. Without trying to start a full thread in this discussion about how to migrate components, migrating some tenants of a multi tenant aware component might also be more difficult and you have to take that into account as well. > @Bram: for the cons you state "I got OOM at 500 tenant with 256M memory in > poc setup": what causes this OOM? It seems to me as a pretty simple > application which only 500 tenants (and sure, only 256M memory). It is the > overhead of 500 containers or something else? My guess would be it is, but you can quite easily run a benchmark that instantiates and starts 500 completely empty containers and measure memory consumption. Only half a megabyte of memory for a container does not sound too bad. > What would be the memory footprint for BC assuming all BC specific bundles > are tenant unaware, assuming BC also needs memory for caching purposes. I'll leave that one to Bram, but in general I think it's a hard question to answer because I cannot imagine that it's independent of how BC is being used. Unless you're talking about code size, not runtime consumption. > How well does it scale when I have a small application (eg openID server) > with lots of tenants. You'll have the overhead of each OSGi framework instance (which you can measure as mentioned above). Other than that, there should not be much overhead if each tenant uses its own data. I mean you might be able to more effectively share resources if you have a multi tenant aware implementation (one pool for every shared resource instead of lots of smaller pools) but that's about it. > A more practical question: assume I have a tenant unaware service with a REST > interface. I assume the REST is received by the tenant aware HTTP service > (which listens to a certain port) in the tenant aware container. How is this > call forwarded to the tenant unaware service? I don't think we currently have > a mechanism to go from tenant aware to tenant unaware, where you need to know > in what container the tenant lives. So far we only talked about OSGi services, and there we use remote services to publish the right tenant service into its container. For REST services, there is no such thing as a service registry, so we cannot discover them. That means we need to somehow configure them. In this case it means we need to map each tenant onto the appropriate REST URL. Greetings, Marcel _______________________________________________ Amdatu-developers mailing list Amdatu-developers at amdatu.org http://lists.amdatu.org/mailman/listinfo/amdatu-developers
