Re: [Openstack] Pondering multi-tenant needs in nova.
inline: On 2/8/11 8:30 AM, Vishvananda Ishaya wrote: This thread is enormous, so I'm I'm going to briefly summarize the two options as I see them: 1. Project Id is an opaque string, and it simply represents some kind of collection of users. It is the responsibility of external systems (authn, authz, billing, and monitoring) to define what the string means, and what the relationships between the different groups and objects are. Nova needs a few plug-in points to authn and authz, but the logic relating these projects to users happens external to the nova code. Networking concerns are isolated to the project level. Pros: * Almost no changes to existing code (supporting multiple networks per project isonly necessary if we want to support multi-tenancy in vlan mode) * project code is simple and unencumbered (most small deployments don't need multi-tenancy) This is not a Pro, this is a Con. We must design OpenStack around the largest possible deployment while keeping the small ones in mind, not the other way around. Furthermore about multi-tenancy: a) Why are we thinking of multi-tenancy as an optional component? It should be multi-tenancy by nature in that any tenant can have a sub-tenant. At some level don't we want a supreme tenant that can control all others? In which case all it is is tenants and sub-tenants. Perhaps I am simply off base here in the relations between tenants-Accounts-...-Clouds(vms, network, storage services, etc., etc.). Is this clearly defined now? If I am off base would someone be so kind and educate this poor sole by pointing me to a link :) Regards, Aimon Cons: * pushing a lot of potentially useful code into external systems * potential for lack of code sharing between external sytsems 2. We change the project code into a more general concept of groups. Groups can contain other groups, and users and groups can be members of multiple groups. This would mirror the possibilities available in ldap. Pros: * Greater flexibility of implementation. * Group implementation code is in one place, minimizing different implementations per component Cons: * Much more complexity in the nova code. * The greater complexity/flexibility won't be needed for smaller deployments. Both of these options seem viable to me, but I'm actually leaning toward adding flexible groups into nova proper. A complete authentication system IMO needs to support, flexible groups. If we increase the flexibility of nova in this regard, it gives us a springboard to breaking out authz into a more complete separate service. I like this more than rewriting the entire thing from scratch as a completely new component. Vish On Feb 8, 2011, at 8:11 AM, Jay Pipes wrote: Hey Paul, yeah, see what happens when you take a little time away from email? ;P So, I'm satisfied that I've highlighted the trade-offs that come along with Nova not inherently understanding the relationships between accounts. Having an external system understand these account relationships is fine, and the posters on this thread have done a good job explaining the benefits that come along with federating the responsibility to an external plugin/service, but there are some performance issues that come along with it. However, as long as these inefficiencies are known, I'm satisfied. :) Cheers! jay On Mon, Feb 7, 2011 at 11:37 PM, Paul Voccio paul.voc...@rackspace.com wrote: Woah, seems I missed a lot by not being around email today. I was a bit confused at to why we would want to have nova trackif an account was being used by a reseller. In digging back through the blueprint associated with this, it seems the idea is for the operator (in this case Rackspace, but whoever) of Nova should track the idea of a reseller and accounts associated with that reseller. Nova itself would still retain the idea of a single account and the resources associated with that account. I guess this doesn't feel any different than another managed service provider who is doing add-on business on top of a amazon, rackspace, linode or other cloud business. https://blueprints.launchpad.net/nova/+spec/multi-tenant-accounting, specifically: http://wiki.openstack.org/openstack-accounting?action=AttachFiledo=viewta rget=accounts.pdf While an operator *could* implement the account id as an arbitrary string and map inefficient queries to it as Jay mentions, I'm not sure they would (or even should). Jay -- I think I understand your concerns, but are you suggesting we implement the idea layer of resellers into Nova? Did I miss the point? Sorry if I'm late to the party on this one. Pvo On 2/7/11 8:20 PM, Eric Day e...@oddments.org wrote: On Mon, Feb 07, 2011 at 08:50:58PM -0500, Jay Pipes wrote: Eric, you and I have a database background. I know you understand that this: Of course, but the first pair of queries is not as bad as a query for every entity ID
Re: [Openstack] Pondering multi-tenant needs in nova.
This is slightly tangental but assume there are environments running both Swift and Nova with a set of customers using both compute and object store services. For example, user johnsmith logs into his future openstack portal and sees both his VMs and his files and containers. Even though these two underlying services are separate technologies its likely we will be be creating facilities to integrate them for various functions. In the future when I backup or save my VM, those disk image archives should be uploaded to my swift account in some special container so that that disk space used can be billed accordingly against johnsmith. I may then in the future spawn a new VM and the source of the image would be a swift container:object. I may also upload an ISO to swift and create new images from scratch by having kvm/xen boot from the ISO and I use ajaxterm to install the OS, etc. I just list these as possibilities to strengthen my case that there could be a 1 to 1 relationship to swift and nova accounts. With this relationship, I will then also need facilities to impersonate or carry out API commands on behalf johnsmith from nova to swift. How does nova send something to swift on half of specific account without storing the actual credentials of the swift users in nova? The simple answer could just be a private set of swift proxies with custom auth middleware, but we still need to track this 1 to 1 relationship of accounts, or we need to unify auth/project/account layers? Maybe somebody has already thought this out and knows a simpler way to do it? -JC On Feb 8, 2011, at 10:15 AM, Paul Voccio wrote: On 2/8/11 10:30 AM, Vishvananda Ishaya vishvana...@gmail.com wrote: This thread is enormous, so I'm I'm going to briefly summarize the two options as I see them: 1. Project Id is an opaque string, and it simply represents some kind of collection of users. It is the responsibility of external systems (authn, authz, billing, and monitoring) to define what the string means, and what the relationships between the different groups and objects are. Nova needs a few plug-in points to authn and authz, but the logic relating these projects to users happens external to the nova code. Networking concerns are isolated to the project level. Pros: * Almost no changes to existing code (supporting multiple networks per project isonly necessary if we want to support multi-tenancy in vlan mode) Won't we still have to support multiple networks per project in flat networking as well? You could go into multiple zones, supported by different networking gear that will need different networks routed to them. * project code is simple and unencumbered (most small deployments don't need multi-tenancy) Cons: * pushing a lot of potentially useful code into external systems * potential for lack of code sharing between external sytsems 2. We change the project code into a more general concept of groups. Groups can contain other groups, and users and groups can be members of multiple groups. This would mirror the possibilities available in ldap. Pros: * Greater flexibility of implementation. * Group implementation code is in one place, minimizing different implementations per component Cons: * Much more complexity in the nova code. * The greater complexity/flexibility won't be needed for smaller deployments. This seems like it is an issue only larger deployments of Nova will encounter. Trying to guess how someone will try to implement and bill for resellers and groups and which group should have access to another group's resources seems almost out of scope at the moment. Both of these options seem viable to me, but I'm actually leaning toward adding flexible groups into nova proper. A complete authentication system IMO needs to support, flexible groups. If we increase the flexibility of nova in this regard, it gives us a springboard to breaking out authz into a more complete separate service. I like this more than rewriting the entire thing from scratch as a completely new component. Vish On Feb 8, 2011, at 8:11 AM, Jay Pipes wrote: Hey Paul, yeah, see what happens when you take a little time away from email? ;P So, I'm satisfied that I've highlighted the trade-offs that come along with Nova not inherently understanding the relationships between accounts. Having an external system understand these account relationships is fine, and the posters on this thread have done a good job explaining the benefits that come along with federating the responsibility to an external plugin/service, but there are some performance issues that come along with it. However, as long as these inefficiencies are known, I'm satisfied. :) Cheers! jay On Mon, Feb 7, 2011 at 11:37 PM, Paul Voccio paul.voc...@rackspace.com wrote: Woah, seems I missed a lot by not being around email today. I was a bit confused at to why we
Re: [Openstack] Pondering multi-tenant needs in nova.
+1 From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of Eric Day [e...@oddments.org] Sent: Monday, February 07, 2011 1:35 AM To: Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I disagree with your disagreement. :) When we have string based ID's like this, it doesn't need to translate directly into a varchar column for operations. First, auth data may not be stored as SQL at all for some systems and could be broken out into key/value pairs with some indexed. It could also translate directly into a LDAP hierarchy which can be tuned to be very efficient. For SQL-based auth storage, this could remain pluggable according to the how the organization creates the string. For example, the string may be broken out into parts for the auth lookup and mapped to various columns/tables to search/join together efficiently. The examples below where you have 'X-Y-Z' format is assuming a certain structure/layout, but I think we should treat the string as completely opaque outside the auth plugins and let the auth plugins perform optimized translation and lookup into auth and access control objects that are used in the rest of the code for the various projects (much like it is today). -Eric On Sun, Feb 06, 2011 at 09:57:56AM -0500, Jay Pipes wrote: Strongly disagree, but nicely, of course :) I'll disagree by showing you an example of why not having a queryable org model is problematic: Let's say we go ahead and do what Glen suggests and have a string account ID that is then attached to the user in a one to many relationship. In SQL (MySQL variant below), this is represented as so: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Proposed accounts table, with string based tag-like account identifier: CREATE TABLE accounts ( id VARCHAR(255) NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); Now let's say that we store account IDs like this: enterprise-dept-milestone. How would we get all accounts in Enterprise X? Easy, and efficiently: SELECT id FROM accounts WHERE id LIKE X% How would we get all accounts in Enterprise X and Dept Y? Again, it would be easy and efficient: SELECT id FROM accounts WHERE id LIKE X-Y-% But, what happens if multiple departments can work on the same milestone (a common requirement)? How do we query for all accounts in Enterprise X and Milestone Z? The SQL would be horrific, and with millions of records, would bog the reporting system down (trust me): SELECT id FROM accounts WHERE id LIKE X%-%-%Z. The above query would force a full table scan across the entire accounts table. An organization like Rackspace would theoretically have millions of account records (# customers + (# customers X #customer projects) + (# resellers X # reseller customers) + (# reseller customers X # reseller customer projects)) The simpler query of getting all accounts working on a milestone now becomes equally inefficient: SELECT id FROM accounts WHERE if LIKE %-Z The above query also has the side-effect of introducing subtle bugs when, and this will happen because of Murphy's law, accounts called Rackspace-Accounting and Rackspace-IT-Accounting are created. Now, the account for the accounting department and the IT department's Accounting milestone are erroneously returned. While it may seem nice and easy to put string-based, loose tags into the system, this decision is extremely difficult to reverse when made, and it leads to inefficiencies in the querying of the system and subtle query bugs as noted above. A more robust way of structuring the schema is like so, again in the MySQL SQL variant: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Organizations are collections of users that can contain other organizations CREATE TABLE organization ( id INT NOT NULL NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, parent INT NULL, # Adjacency list model enables efficient child and parent lookups left INT NULL, # left and right enable the nested sets model that enables right INT NULL, # equally efficient lookups of more complex relationships FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); The above structure can accomodate both simple (get my immediate parent or immediate children) queries and complex queries (get ALL my children, aggregate querying across the entire tree or subtrees) and do so efficiently. The query API interface that we expose via
Re: [Openstack] Pondering multi-tenant needs in nova.
I am really proposing to get rid of the projects table entirely and consume it with the SQL structure I proposed that can handle nested organizational units. On Sun, Feb 6, 2011 at 6:09 PM, Brian Schott bfsch...@gmail.com wrote: Going back to the original question, it seems that projects has all of the necessary authentication, quota, audit hooks, and role-based management of granting users access to projects to support multi-tennant management, but the issue is that you can't have multiple projects in a flat network? If you create an accounts (opaque string or some arbitrary data structure doesn't matter) don't you have to create all of the same quota/role functionality as projects without the network? Or is this just an opaque filter on all of these table views? Or are we talking about Accounts as a box to the LEFT of Users that groups multiple users into a single billing domain? http://nova.openstack.org/adminguide/managing.projects.html Brian Schott bfsch...@gmail.com On Feb 6, 2011, at 3:59 PM, Sandy Walsh wrote: I think, but I'm not sure, that Glen is suggesting to just keep a URI for the enterprise element. This might map to a database, LDAP, etc. and it might be internal to Nova or external (likely). One plug-in for tentant structure might from a database, in which your proposal would be perfect (parsing the key as the tree to fetch). But another could be LDAP, where you would ask it for the parents/siblings/children. The plugin interface: Who is my parent? Who are my children? Can Customer A see Customer B? Who are the clients of Reseller X? Who do I bill? Where does the audit log come from? ... this seems to be where the real magic has to happen. Unless I missed something? -Sandy From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of Jay Pipes [jaypi...@gmail.com] Sent: Sunday, February 06, 2011 10:57 AM To: John Purrier Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. Strongly disagree, but nicely, of course :) I'll disagree by showing you an example of why not having a queryable org model is problematic: Let's say we go ahead and do what Glen suggests and have a string account ID that is then attached to the user in a one to many relationship. In SQL (MySQL variant below), this is represented as so: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Proposed accounts table, with string based tag-like account identifier: CREATE TABLE accounts ( id VARCHAR(255) NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); Now let's say that we store account IDs like this: enterprise-dept-milestone. How would we get all accounts in Enterprise X? Easy, and efficiently: SELECT id FROM accounts WHERE id LIKE X% How would we get all accounts in Enterprise X and Dept Y? Again, it would be easy and efficient: SELECT id FROM accounts WHERE id LIKE X-Y-% But, what happens if multiple departments can work on the same milestone (a common requirement)? How do we query for all accounts in Enterprise X and Milestone Z? The SQL would be horrific, and with millions of records, would bog the reporting system down (trust me): SELECT id FROM accounts WHERE id LIKE X%-%-%Z. The above query would force a full table scan across the entire accounts table. An organization like Rackspace would theoretically have millions of account records (# customers + (# customers X #customer projects) + (# resellers X # reseller customers) + (# reseller customers X # reseller customer projects)) The simpler query of getting all accounts working on a milestone now becomes equally inefficient: SELECT id FROM accounts WHERE if LIKE %-Z The above query also has the side-effect of introducing subtle bugs when, and this will happen because of Murphy's law, accounts called Rackspace-Accounting and Rackspace-IT-Accounting are created. Now, the account for the accounting department and the IT department's Accounting milestone are erroneously returned. While it may seem nice and easy to put string-based, loose tags into the system, this decision is extremely difficult to reverse when made, and it leads to inefficiencies in the querying of the system and subtle query bugs as noted above. A more robust way of structuring the schema is like so, again in the MySQL SQL variant: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Organizations are collections of users
Re: [Openstack] Pondering multi-tenant needs in nova.
: SELECT id FROM accounts WHERE if LIKE %-Z The above query also has the side-effect of introducing subtle bugs when, and this will happen because of Murphy's law, accounts called Rackspace-Accounting and Rackspace-IT-Accounting are created. Now, the account for the accounting department and the IT department's Accounting milestone are erroneously returned. While it may seem nice and easy to put string-based, loose tags into the system, this decision is extremely difficult to reverse when made, and it leads to inefficiencies in the querying of the system and subtle query bugs as noted above. A more robust way of structuring the schema is like so, again in the MySQL SQL variant: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Organizations are collections of users that can contain other organizations CREATE TABLE organization ( id INT NOT NULL NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, parent INT NULL, # Adjacency list model enables efficient child and parent lookups left INT NULL, # left and right enable the nested sets model that enables right INT NULL, # equally efficient lookups of more complex relationships FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); The above structure can accomodate both simple (get my immediate parent or immediate children) queries and complex queries (get ALL my children, aggregate querying across the entire tree or subtrees) and do so efficiently. The query API interface that we expose via Nova (that would be consumed by some reporting/audit/management tools) would therefore not be a serious drain on the databases storing Nova data. More information on the adjacency list and nested sets models are available here: http://en.wikipedia.org/wiki/Nested_set_model http://en.wikipedia.org/wiki/Adjacency_list_model I'd highly recommend this solution as opposed to the seemingly simple tag-based solution that leads to gross querying inefficiencies and subtle bugs. Just my two cents. -jay On Thu, Feb 3, 2011 at 7:38 PM, John Purrier j...@openstack.org wrote: I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run
Re: [Openstack] Pondering multi-tenant needs in nova.
If it's any help, Swift just uses an opaque account string. With Cloud Files we have several resellers each with their own sets of how accounts and users should be structured. We backed away quickly from all that and went the route of here's an account identifier value, it must be unique amongst your accounts for each reseller. This doesn't prevent us from allowing access controls, etc. as the authentication system can return a list of groups (or roles, or w/e, these are opaque strings to us too). Even if Swift had the functionality for mapping org structures, I can't imagine our resellers would want to mirror that over to us, it's hard enough to maintain in one system. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
On Feb 7, 2011, at 8:30 AM, Jay Pipes wrote: What Swift APIs are available for a reseller to query which of its customer accounts have consumed X resources? Or does Swift punt and make the reseller calculate all those things? A bit of both. You can head an account to get get disk usage, object count type stats. But most stats are via log processing after the fact. Does Swift enforce the uniqueness constraint you allude to above? If not, why bother even having the uniqueness constraint? Yes, the reseller creates the Swift account, if it already exists they get a 409 conflict. Again, this has nothing to do with authentication. Nova has a separate users table that relates to the auth system. This, AFAIK, has nothing to do with auth. Okay, okay, no need to get upset. :P They are related in Swift though. For instance, if someone wanted to set up a container that several users could all access they would do that with these opaque strings. They wouldn't create a new group in Swift or some such. Those kind of org structures are outside Swift. Swift just knows allow x to have access and token t auths to groups g. Depends how much they value the ability not to have to constantly query information from the Swift API and then immediately re-structure the data and construct aggregate reports from the raw data after munging it into their own org structure. Sure. You make it sound so evil, but it's not that bad. Account x maps to something on their side and they're done. Someone has to do the aggregate reporting, etc. work. We're working towards an example system that does that, but we're trying to keep it out of the core of Swift if possible. A good reason for this is that we expect a lot of improvements to happen in this area, and likely they'll be applicable to more than just Swift. But, all that said, Nova's mileage may vary. :) Just stating what Swift's been doing if it's any help. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
On Mon, Feb 7, 2011 at 10:33 AM, Greg gh...@rackspace.com wrote: On Feb 7, 2011, at 8:30 AM, Jay Pipes wrote: What Swift APIs are available for a reseller to query which of its customer accounts have consumed X resources? Or does Swift punt and make the reseller calculate all those things? A bit of both. You can head an account to get get disk usage, object count type stats. But most stats are via log processing after the fact. OK, but can you head all the accounts under another account? Or is that done via log processing? Does Swift enforce the uniqueness constraint you allude to above? If not, why bother even having the uniqueness constraint? Yes, the reseller creates the Swift account, if it already exists they get a 409 conflict. Gotcha, thx. Again, this has nothing to do with authentication. Nova has a separate users table that relates to the auth system. This, AFAIK, has nothing to do with auth. Okay, okay, no need to get upset. :P Hehe, I'm not upset, no worries :) Just trying to ensure that various perspectives are explored before a final decision on this is made. They are related in Swift though. For instance, if someone wanted to set up a container that several users could all access they would do that with these opaque strings. They wouldn't create a new group in Swift or some such. Those kind of org structures are outside Swift. Swift just knows allow x to have access and token t auths to groups g. Gotcha. Yes, in Nova authentication is handled separately, with the users table holding the identifier(s) that are used in authentication. For authorization, the different APIs use different models IIRC. For EC2, the security_groups and security_group_roles tables hold some of this information. There is also the ability to use an external store such as LDAP authorization too, IIRC. Vishy or Devin, feel free to correct me here if I'm off base. Been a while since I looked at that piece of Nova. Depends how much they value the ability not to have to constantly query information from the Swift API and then immediately re-structure the data and construct aggregate reports from the raw data after munging it into their own org structure. Sure. You make it sound so evil, but it's not that bad. Account x maps to something on their side and they're done. Someone has to do the aggregate reporting, etc. work. We're working towards an example system that does that, but we're trying to keep it out of the core of Swift if possible. A good reason for this is that we expect a lot of improvements to happen in this area, and likely they'll be applicable to more than just Swift. But, all that said, Nova's mileage may vary. :) Just stating what Swift's been doing if it's any help. Understood, and appreciated. :) Like I said, just voicing a dissenting opinion to ensure the discussion contains opposing viewpoints and that all perspectives are explored. Very much appreciated your input from the Swift perspective and experience. :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile 210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https
Re: [Openstack] Pondering multi-tenant needs in nova.
think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile 210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message
Re: [Openstack] Pondering multi-tenant needs in nova.
...@openstack.org wrote: I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile 210-441-0965 google voice
Re: [Openstack] Pondering multi-tenant needs in nova.
milestone are erroneously returned. While it may seem nice and easy to put string-based, loose tags into the system, this decision is extremely difficult to reverse when made, and it leads to inefficiencies in the querying of the system and subtle query bugs as noted above. A more robust way of structuring the schema is like so, again in the MySQL SQL variant: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Organizations are collections of users that can contain other organizations CREATE TABLE organization ( id INT NOT NULL NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, parent INT NULL, # Adjacency list model enables efficient child and parent lookups left INT NULL, # left and right enable the nested sets model that enables right INT NULL, # equally efficient lookups of more complex relationships FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); The above structure can accomodate both simple (get my immediate parent or immediate children) queries and complex queries (get ALL my children, aggregate querying across the entire tree or subtrees) and do so efficiently. The query API interface that we expose via Nova (that would be consumed by some reporting/audit/management tools) would therefore not be a serious drain on the databases storing Nova data. More information on the adjacency list and nested sets models are available here: http://en.wikipedia.org/wiki/Nested_set_model http://en.wikipedia.org/wiki/Adjacency_list_model I'd highly recommend this solution as opposed to the seemingly simple tag-based solution that leads to gross querying inefficiencies and subtle bugs. Just my two cents. -jay On Thu, Feb 3, 2011 at 7:38 PM, John Purrier j...@openstack.org wrote: I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple
Re: [Openstack] Pondering multi-tenant needs in nova.
(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Organizations are collections of users that can contain other organizations CREATE TABLE organization ( id INT NOT NULL NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, parent INT NULL, # Adjacency list model enables efficient child and parent lookups left INT NULL, # left and right enable the nested sets model that enables right INT NULL, # equally efficient lookups of more complex relationships FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); The above structure can accomodate both simple (get my immediate parent or immediate children) queries and complex queries (get ALL my children, aggregate querying across the entire tree or subtrees) and do so efficiently. The query API interface that we expose via Nova (that would be consumed by some reporting/audit/management tools) would therefore not be a serious drain on the databases storing Nova data. More information on the adjacency list and nested sets models are available here: http://en.wikipedia.org/wiki/Nested_set_model http://en.wikipedia.org/wiki/Adjacency_list_model I'd highly recommend this solution as opposed to the seemingly simple tag-based solution that leads to gross querying inefficiencies and subtle bugs. Just my two cents. -jay On Thu, Feb 3, 2011 at 7:38 PM, John Purrierj...@openstack.org wrote: I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlendevin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragonmdra...@rackspace.com Cc:openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default
Re: [Openstack] Pondering multi-tenant needs in nova.
table. An organization like Rackspace would theoretically have millions of account records (# customers + (# customers X #customer projects) + (# resellers X # reseller customers) + (# reseller customers X # reseller customer projects)) The simpler query of getting all accounts working on a milestone now becomes equally inefficient: SELECT id FROM accounts WHERE if LIKE %-Z The above query also has the side-effect of introducing subtle bugs when, and this will happen because of Murphy's law, accounts called Rackspace-Accounting and Rackspace-IT-Accounting are created. Now, the account for the accounting department and the IT department's Accounting milestone are erroneously returned. While it may seem nice and easy to put string-based, loose tags into the system, this decision is extremely difficult to reverse when made, and it leads to inefficiencies in the querying of the system and subtle query bugs as noted above. A more robust way of structuring the schema is like so, again in the MySQL SQL variant: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Organizations are collections of users that can contain other organizations CREATE TABLE organization ( id INT NOT NULL NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, parent INT NULL, # Adjacency list model enables efficient child and parent lookups left INT NULL, # left and right enable the nested sets model that enables right INT NULL, # equally efficient lookups of more complex relationships FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); The above structure can accomodate both simple (get my immediate parent or immediate children) queries and complex queries (get ALL my children, aggregate querying across the entire tree or subtrees) and do so efficiently. The query API interface that we expose via Nova (that would be consumed by some reporting/audit/management tools) would therefore not be a serious drain on the databases storing Nova data. More information on the adjacency list and nested sets models are available here: http://en.wikipedia.org/wiki/Nested_set_model http://en.wikipedia.org/wiki/Adjacency_list_model I'd highly recommend this solution as opposed to the seemingly simple tag-based solution that leads to gross querying inefficiencies and subtle bugs. Just my two cents. -jay On Thu, Feb 3, 2011 at 7:38 PM, John Purrier j...@openstack.org wrote: I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization
Re: [Openstack] Pondering multi-tenant needs in nova.
FROM accounts WHERE id LIKE X% How would we get all accounts in Enterprise X and Dept Y? Again, it would be easy and efficient: SELECT id FROM accounts WHERE id LIKE X-Y-% But, what happens if multiple departments can work on the same milestone (a common requirement)? How do we query for all accounts in Enterprise X and Milestone Z? The SQL would be horrific, and with millions of records, would bog the reporting system down (trust me): SELECT id FROM accounts WHERE id LIKE X%-%-%Z. The above query would force a full table scan across the entire accounts table. An organization like Rackspace would theoretically have millions of account records (# customers + (# customers X #customer projects) + (# resellers X # reseller customers) + (# reseller customers X # reseller customer projects)) The simpler query of getting all accounts working on a milestone now becomes equally inefficient: SELECT id FROM accounts WHERE if LIKE %-Z The above query also has the side-effect of introducing subtle bugs when, and this will happen because of Murphy's law, accounts called Rackspace-Accounting and Rackspace-IT-Accounting are created. Now, the account for the accounting department and the IT department's Accounting milestone are erroneously returned. While it may seem nice and easy to put string-based, loose tags into the system, this decision is extremely difficult to reverse when made, and it leads to inefficiencies in the querying of the system and subtle query bugs as noted above. A more robust way of structuring the schema is like so, again in the MySQL SQL variant: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Organizations are collections of users that can contain other organizations CREATE TABLE organization ( id INT NOT NULL NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, parent INT NULL, # Adjacency list model enables efficient child and parent lookups left INT NULL, # left and right enable the nested sets model that enables right INT NULL, # equally efficient lookups of more complex relationships FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); The above structure can accomodate both simple (get my immediate parent or immediate children) queries and complex queries (get ALL my children, aggregate querying across the entire tree or subtrees) and do so efficiently. The query API interface that we expose via Nova (that would be consumed by some reporting/audit/management tools) would therefore not be a serious drain on the databases storing Nova data. More information on the adjacency list and nested sets models are available here: http://en.wikipedia.org/wiki/Nested_set_model http://en.wikipedia.org/wiki/Adjacency_list_model I'd highly recommend this solution as opposed to the seemingly simple tag-based solution that leads to gross querying inefficiencies and subtle bugs. Just my two cents. -jay On Thu, Feb 3, 2011 at 7:38 PM, John Purrier j...@openstack.org wrote: I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800
Re: [Openstack] Pondering multi-tenant needs in nova.
On Mon, Feb 7, 2011 at 8:24 PM, Jorge Williams jorge.willi...@rackspace.com wrote: On Feb 7, 2011, at 5:43 PM, Jay Pipes wrote: What if I don't want to get my servers only? What if I want to list another organization's servers, and that organization's child organizations' servers? That sort of information and those sort of quires fall within the realm of configuration management. So for any organization or customer or whatever you should be able to ask: What resources belong to that customer/organization? (Note just compute resources, but object store resources, or whatever) What resources have been requested and are actively being provisioned? etc. There should be a service (the configuration management service) that's there just for answering those sort of questions -- the service shouldn't necessarily have to query nova or swift directly -- instead the underlying configuration management database can be populated by listening to events from swift and nova. The shape of the CMDB and what queries it's optimized to process will vary from one deployment to another. Yes, I understand what you, Eric, Monsyne, Greg, John, and others have been saying :) I am just saying that if the decision is to federate the responsibility of determining account relationships to an external service, that we should understand that the efficiency of certain queries is affected, and that API operations must, by nature, be restricted to a simpler set of operations -- i.e. get me the list of servers attached to accounts X, Y, and Z, as opposed to get me all servers attached to the X account and any of X's child accounts. The latter example, there must instead be split into two requests: one to the auth service for get me all accounts, inclusively, of account X and all of X's child accounts, and another request to Nova for get me all the servers attached to accounts X, Y, and Z, where X, Y, Z is the list of accounts returned by the first request. I'm not saying the federated auth/CMDB/whatever service is not a good idea nor that it will not work. I want people to understand and acknowledge the tradeoffs involved. Cheers, jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
On Feb 7, 2011, at 7:58 PM, Jay Pipes wrote: I'm not saying the federated auth/CMDB/whatever service is not a good idea nor that it will not work. I want people to understand and acknowledge the tradeoffs involved. Acknowledged? :) [Hehe, damned programmers will never agree on nearly anything; probably even this statement. :)] ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
Strongly disagree, but nicely, of course :) I'll disagree by showing you an example of why not having a queryable org model is problematic: Let's say we go ahead and do what Glen suggests and have a string account ID that is then attached to the user in a one to many relationship. In SQL (MySQL variant below), this is represented as so: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Proposed accounts table, with string based tag-like account identifier: CREATE TABLE accounts ( id VARCHAR(255) NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); Now let's say that we store account IDs like this: enterprise-dept-milestone. How would we get all accounts in Enterprise X? Easy, and efficiently: SELECT id FROM accounts WHERE id LIKE X% How would we get all accounts in Enterprise X and Dept Y? Again, it would be easy and efficient: SELECT id FROM accounts WHERE id LIKE X-Y-% But, what happens if multiple departments can work on the same milestone (a common requirement)? How do we query for all accounts in Enterprise X and Milestone Z? The SQL would be horrific, and with millions of records, would bog the reporting system down (trust me): SELECT id FROM accounts WHERE id LIKE X%-%-%Z. The above query would force a full table scan across the entire accounts table. An organization like Rackspace would theoretically have millions of account records (# customers + (# customers X #customer projects) + (# resellers X # reseller customers) + (# reseller customers X # reseller customer projects)) The simpler query of getting all accounts working on a milestone now becomes equally inefficient: SELECT id FROM accounts WHERE if LIKE %-Z The above query also has the side-effect of introducing subtle bugs when, and this will happen because of Murphy's law, accounts called Rackspace-Accounting and Rackspace-IT-Accounting are created. Now, the account for the accounting department and the IT department's Accounting milestone are erroneously returned. While it may seem nice and easy to put string-based, loose tags into the system, this decision is extremely difficult to reverse when made, and it leads to inefficiencies in the querying of the system and subtle query bugs as noted above. A more robust way of structuring the schema is like so, again in the MySQL SQL variant: # Our existing users table: CREATE TABLE users ( id VARCHAR(255) NOT NULL PRIMARY KEY, access_key VARCHAR(255) NOT NULL, secret_key VARCHAR(255) NOT NULL, is_admin TINYINT NOT NULL DEFAULT 0 ); # Organizations are collections of users that can contain other organizations CREATE TABLE organization ( id INT NOT NULL NOT NULL PRIMARY KEY, user_id VARCHAR(255) NOT NULL, parent INT NULL, # Adjacency list model enables efficient child and parent lookups left INT NULL, # left and right enable the nested sets model that enables right INT NULL, # equally efficient lookups of more complex relationships FOREIGN KEY fk_users (user_id) REFERENCES users (id) ); The above structure can accomodate both simple (get my immediate parent or immediate children) queries and complex queries (get ALL my children, aggregate querying across the entire tree or subtrees) and do so efficiently. The query API interface that we expose via Nova (that would be consumed by some reporting/audit/management tools) would therefore not be a serious drain on the databases storing Nova data. More information on the adjacency list and nested sets models are available here: http://en.wikipedia.org/wiki/Nested_set_model http://en.wikipedia.org/wiki/Adjacency_list_model I'd highly recommend this solution as opposed to the seemingly simple tag-based solution that leads to gross querying inefficiencies and subtle bugs. Just my two cents. -jay On Thu, Feb 3, 2011 at 7:38 PM, John Purrier j...@openstack.org wrote: I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id
Re: [Openstack] Pondering multi-tenant needs in nova.
@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
recommend this solution as opposed to the seemingly simple tag-based solution that leads to gross querying inefficiencies and subtle bugs. Just my two cents. -jay On Thu, Feb 3, 2011 at 7:38 PM, John Purrier j...@openstack.org wrote: I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile 210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed
Re: [Openstack] Pondering multi-tenant needs in nova.
Salvatore is right here. The key point is that the limits of standard L2 + VLANs aren't just about the number of bits in the VLAN tag: even if you buy gear that supports more VLANs in hardware (including Q-in-Q), more important are the issues with broadcast scalability, MAC table size, inability to L2 to leverage multiple active paths, inability to fail-over quickly to alternate paths, and inability to span an L3 backbone that joins disparate datacenters. We built open vswitch to have L2-in-L3 tunneling capabilities exactly because of these limitations and will be working with the openstack team to make sure open vswitch can be used in openstack whether you prefer to us VLAN or more advanced tunneling options. Dan On Fri, Feb 4, 2011 at 2:02 AM, Salvatore Orlando salvatore.orla...@eu.citrix.com wrote: Hi, I agree with Aimon and Patrick that a VLAN per customer would be a good fit only for small deployments. Moreover, if I'm not wrong, datacentres are typically organized in small layer-2 PoD in the Access layer interconnected by L3 routers (I'm referring to the Cisco Data Center Infrastructure design guide); to the best of my knowledge L3 routers do not forward VLAN tags. In the access layer, PoDs are typically small in order to limit the impact of L2 broadcast traffic, and, I believe, to make spanning tree working more efficiently. One thing we might consider in order to overcome this limit is to employ L3 tunnelling protocols for VM traffic; in my opinion this might have several benefits: - No need to perform any configuration, such as setting ports on trunking mode, on physical network switches: they would not see VLAN-tagged frames, but only IP traffic; - MAC forwarding tables on physical network switches will hardly be saturated, even when using 'cheap' switches, as the only MACs they would see are the endpoints of the tunnels; - L2 broadcast domain will still be contained in size, even if distributed over several PoD in the same data centre and potentially over several data centres. One (potentially big) issue with tunnelling protocols is the topology of the network created by the tunnel themselves; it might define traffic bottlenecks, or even contain loops. I have been using Open vSwitch for a few months now, which has been mentioned on several blueprints in the nova project. It supports GRE tunnels and VLAN tagging (though AFAIK only 802.1Q); it could potentially be a good fit for implementing multi-tenant support for large deployments. Open vSwitch currently runs on both Xen and KVM hypervisors; however it is not supported in ESX, and (If I'm not wrong) in Hyper-V. Regards, Salvatore -Original Message- From: openstack-bounces+salvatore.orlando=eu.citrix.com@ lists.launchpad.net [mailto:openstack-bounces+salvatore.orlandoopenstack-bounces%2Bsalvatore.orlando =eu.citrix@lists.launchpad.net] On Behalf Of Aimon Bustardo Sent: 04 February 2011 07:14 To: Patrick Ancillotti Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. Hi Patrick, that would be great if you would go into details. I am most interested in this as it directly effects our cloud platforms adoption of OpenStack and the subsequent networking blueprint we have designed (trying to get a hold of the networking blueprint Ewan is leading also). Thank you, Aimon On 2/3/11 10:18 PM, Patrick Ancillotti wrote: Aimon, You're correct of course for simply defining a customer per VLAN as realistically we wouldn't hit 16M+ customers in any regional area as it stands today ;) but there are other issues with QinQ at large scale, especially with Layer 2 domains of the size that we're envisaging in the long run... many of which we can go into detail if you want? For those interested in QinQ : http://en.wikipedia.org/wiki/802.1QinQ Thanks, Patrick On 3 Feb 2011, at 23:45, Aimon Bustardo wrote: Hi Patrick, by using 802.1QiQ aren't we increasing the total VLAN count to 4096*4096 (16277216) possible VLANS, each with the possibility of using the same /8 network? In which case he in table storage of MACs is the limiting factor. Aimon On 2/3/11 8:03 PM, Patrick Ancillotti wrote: Hey Guys, I think Paul may have gotten a bit mixed up between VLAN and CAM tables on switches. The VLAN part of an ethernet frame is 12 bits (0 - 4095) which limits it accordingly. CAM tables however are a limit within switching gear that lists the MAC addresses and their respective source and destination ports. The lower end Cisco switches like the 2960 class switches have a limit of around 8000 MAC addresses, whereas others higher in the stack such as the 4948E class have limits upwards of 55000 MAC addresses, although some vendors have CAM tables in the hundreds of thousands for even their mid range switches. That said, VLAN's are extremely limited, even with QinQ (VLANs within VLANs) when you're
Re: [Openstack] Pondering multi-tenant needs in nova.
Hi Monsyne, it's a very interesting topic and I'm curious about the reason why you are using the Flat Networking set up. From the conversations in other threads it seems the Service Providers prefer different networking approaches: VLAN oriented basically. Regards Diego - Diego Parrilla nubeblog.com | nubeb...@nubeblog.com | twitter.com/nubeblog +34 649 94 43 29 On Thu, Feb 3, 2011 at 2:37 AM, Monsyne Dragon mdra...@rackspace.com wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile 210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
2011/2/3 Diego Parrilla Santamaría diego.parrilla.santama...@gmail.com: it's a very interesting topic and I'm curious about the reason why you are using the Flat Networking set up. From the conversations in other threads it seems the Service Providers prefer different networking approaches: VLAN oriented basically. I don't think this is something that Monsyne has any control over ;) May want to ask Rackspace networking engineers that question! -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
Diego, Due to our networking topology, having a vlan per customer isn't really feasible. Most switches are limited at 4k or 8k or even 32k. With more customers than these switches can reasonably accommodate, having a single vlan per customer either limits the portability within a cloud or limits the scale at which you can build your cloud. Open vSwitch will alleviate some of this pain, but until we get it in XenServer, we're somewhat stuck on flat networking. Paul On 2/3/11 4:20 AM, Diego Parrilla Santamaría diego.parrilla.santama...@gmail.com wrote: Hi Monsyne, it's a very interesting topic and I'm curious about the reason why you are using the Flat Networking set up. From the conversations in other threads it seems the Service Providers prefer different networking approaches: VLAN oriented basically. Regards Diego - Diego Parrilla nubeblog.com | nubeb...@nubeblog.com | twitter.com/nubeblog +34 649 94 43 29 On Thu, Feb 3, 2011 at 2:37 AM, Monsyne Dragon mdra...@rackspace.com wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. +1 From: Devin Carlen devin.car...@gmail.commailto:devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.commailto:mdra...@rackspace.com Cc: openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.commailto:ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
Would a less structured approach work to make billing/reporting more flexible for the community? I was thinking something along the lines of tags. With an arbitrary set of tags, instances could have a tag saying they belong to a certain organisation and/or an enterprise. Reports could be generated collecting the number of instances with a specific tag and it would allow for potentially arbitrary billing patterns. I'll admit to being surprised at myself advocating for arbitrary tags, but having worked on billing systems in the past...the last thing I'd like is for a rigid hierarchy to be put in, because right after it's done it's inevitable that another part of the company wants to change how customers are billed. B -Original Message- From: Devin Carlen devin.car...@gmail.com Sent: Thursday, February 3, 2011 3:02pm To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pondering multi-tenant needs in nova.
On 2/3/11 2:02 PM, Devin Carlen wrote: We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. Yah, we are trying to avoid getting too deep into into organizational structure with the multitenant stuff atm. At the previous summit we were talking about using a pretty flat structure, basically 'accounts' for grouping instances, with a flexible naming scheme, and leave the hierarchy management as being something outside the scope of Nova, which would be taken care of by other systems (since each nova implementer is likely to have very different needs and ideas on how to do that. ). Since the account names would be opaque strings to Nova, if an outside system wanted to use account names like JoeCorp:usdivision:hr:recruiting it could. The main decision we are making here is, do we implement accounts as projects (in which case we need to make projects able to share networks to support our model) or do we add a new account concept (in which case, Rackspace, and others using similar org models, will pretty much ignore the project concept, just using a single 'default' project). On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com mailto:ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp -- -- -Monsyne Dragon work: 210-312-4190 mobile210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error,
Re: [Openstack] Pondering multi-tenant needs in nova.
I think Glen is on the right track here. Having the account_ID be a string with no connotation for Nova allows two benefits: 1) deployments can create the arbitrary organizational models that fit their particular DC, physical, and logical structures, and 2) the Nova code is simpler as the hierarchical concepts do not have any manifestations in the code. Additional benefit includes an easier mapping to the particular identity and authorization system that a deployment chooses to use. John From: openstack-bounces+john=openstack@lists.launchpad.net [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On Behalf Of Glen Campbell Sent: Thursday, February 03, 2011 2:42 PM To: Devin Carlen; Monsyne Dragon Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. I think that this could be done in the current proposal. Specifically, the account_id is an arbitrary string that is generated externally to Nova. You could, for example, easily identify an organizational hierarchy. For example, an accountID could be: enterprise-org-project-milestone From Nova's point of view, it makes no difference, so long as that string is associated with a usage event and regurgitated when reported. The cloud administrator can interpret it however it chooses. For simple organizations, it could be identical to the project_id, or even just blank. The project_id holds the network information, and the account_id tracks the usage and other notifications. There's no good reason for Nova to have to model an organization internally; it certainly wouldn't match all the possible org structures available. From: Devin Carlen devin.car...@gmail.com Date: Thu, 3 Feb 2011 12:02:38 -0800 To: Monsyne Dragon mdra...@rackspace.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Pondering multi-tenant needs in nova. We were just talking about this the other day. We definitely need some kind of further hierarchy. I think a typical kind of use case for multi-tenant could be something like: Enterprise contains Organizations Organizations contain Organizations and Projects Projects contain Instances, etc. In this structure enterprise is just a top level organization. If we structure it this way it would make metering and billing pretty simple. On Feb 2, 2011, at 5:37 PM, Monsyne Dragon wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net
Re: [Openstack] Pondering multi-tenant needs in nova.
Aimon, You're correct of course for simply defining a customer per VLAN as realistically we wouldn't hit 16M+ customers in any regional area as it stands today ;) but there are other issues with QinQ at large scale, especially with Layer 2 domains of the size that we're envisaging in the long run... many of which we can go into detail if you want? For those interested in QinQ : http://en.wikipedia.org/wiki/802.1QinQ Thanks, Patrick On 3 Feb 2011, at 23:45, Aimon Bustardo wrote: Hi Patrick, by using 802.1QiQ aren't we increasing the total VLAN count to 4096*4096 (16277216) possible VLANS, each with the possibility of using the same /8 network? In which case he in table storage of MACs is the limiting factor. Aimon On 2/3/11 8:03 PM, Patrick Ancillotti wrote: Hey Guys, I think Paul may have gotten a bit mixed up between VLAN and CAM tables on switches. The VLAN part of an ethernet frame is 12 bits (0 - 4095) which limits it accordingly. CAM tables however are a limit within switching gear that lists the MAC addresses and their respective source and destination ports. The lower end Cisco switches like the 2960 class switches have a limit of around 8000 MAC addresses, whereas others higher in the stack such as the 4948E class have limits upwards of 55000 MAC addresses, although some vendors have CAM tables in the hundreds of thousands for even their mid range switches. That said, VLAN's are extremely limited, even with QinQ (VLANs within VLANs) when you're talking about 50 000 customers, or even 500 000 customers in the same layer 2 domain, and in many cases using smaller layer 2 domains creates unfortunately small service areas for capacity. For more info: http://en.wikipedia.org/wiki/Virtual_LAN http://en.wikipedia.org/wiki/CAM_Table Thanks, Patrick On 3 Feb 2011, at 07:47, Paul Voccio wrote: Diego, Due to our networking topology, having a vlan per customer isn't really feasible. Most switches are limited at 4k or 8k or even 32k. With more customers than these switches can reasonably accommodate, having a single vlan per customer either limits the portability within a cloud or limits the scale at which you can build your cloud. Open vSwitch will alleviate some of this pain, but until we get it in XenServer, we're somewhat stuck on flat networking. Paul On 2/3/11 4:20 AM, Diego Parrilla Santamaría diego.parrilla.santama...@gmail.com wrote: Hi Monsyne, it's a very interesting topic and I'm curious about the reason why you are using the Flat Networking set up. From the conversations in other threads it seems the Service Providers prefer different networking approaches: VLAN oriented basically. Regards Diego - Diego Parrilla nubeblog.com | nubeb...@nubeblog.com | twitter.com/nubeblog +34 649 94 43 29 On Thu, Feb 3, 2011 at 2:37 AM, Monsyne Dragon mdra...@rackspace.com wrote: I am sorting out some possible implementations for the multi-tenant-accounting blueprint, and the related system-usage-records bp, and I just wanted to run this by anyone interested in such matters. Basically, for multitenant purposes we need to introduce the concept of an 'account' in nova, representing a customer, that basically acts as a label for a group of resources (instances, etc), and for access control (i.e customer a cannot mess w/ customer b's stuff) There was some confusion on how best to implement this, in relation to nova's project concept. Projects are kind of like what we want an account to be, but there are some associations (like one project per network) which are not valid for our flat networking setup. I am kind of straw-polling on which is better here: The options are: 1) Create a new 'account' concept in nova, with an account basically being a subgroup of a project (providers would use a single, default project, with additional projects added if needed for separate brands, or resellers, etc), add in access control per account as well as project, and make sure apis/auth specify account appropriately, have some way for a default account to used (per project) so account doesn't get in the way for non-multitenant users. 2) having account == nova's project, and changing the network associations, etc so projects can support our model (as well as current models). Support for associating accounts (projects) together for resellers, etc would either be delegated outside of nova or added later (it's not a current requirement). In either case, accounts would be identified by name, which would be an opaque string an outside system/person would assign, and could structure to their needs (ie. for associating accounts with common prefixes, etc) -- -- -Monsyne Dragon work: 210-312-4190 mobile210-441-0965 google voice: 210-338-0336 Confidentiality Notice: This e-mail message (including any attached or embedded documents) is
