Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Hi On 6 Oct 2014, at 17:41, Clint Byrum cl...@fewbar.com wrote: We have to be _extremely_ careful in how we manage this. I actually think it has potential to really blow up in our faces. Yes, anything we do here has the potential to be extremely ruinous for operators, but the reality is that any existing TripleO deployment is at pretty severe risk of blowing up because of UIDs/GIDs changing when they update. We need to give people a way to move forward without us merging a patch, and at the same time we need to make sure we provide a consistent set of UIDs for anything people may want to deploy with diskimage-builder. IMO the only desirable option *has* to be that we statically define UIDs and GIDs in the elements, because: 1: Requires no data fragments to be kept safe and fed to subsequent build processes 2: Doesn't do anything dynamic on first boot that could take hours/days 3: Can be thoroughly audited at build time to ensure correctness As you rightly point out though, any existing deployments will definitely be disrupted by this, but as I said above, all we'd be doing there is moving the needle from possible/probable to definite. Since the only leftovers we have from their previous image builds, are the images themselves, we could add the ability for a DIB run to extract IDs from a previous image, but this couldn't be required as a default build option, so we'd still risk existing deployments if they don't notice this feature. We could create a script that would spider an existing cloud and extract its ID mappings, to produce a fragment to feed into future builds, but again we're relying on operators to know that they need to do this. Instead, I agree with Greg's view that this is our fault and we should fix it. We didn't think of this sooner, and as a result, our users are at risk. If we don't entirely fix this ourselves, we will be both expecting them to become aware of this issue and expecting them to do additional work to mitigate it. To that end, I think we should audit all of our elements for use of /mnt/state/ and use the specific knowledge we have of the software they relate to, to build one-time ID migration scripts, which would: 1: Execute before any related services start 2: Compare the now-static ID mappings against known files in /mnt/state 3: chown/chgrp any files/directories that need migrating 4: store a flag file in /mnt/state indicating that this process doesn't need to run again It does mean they have a potentially painfully long update process once, but the result will be a completely stable, static arrangement that will not require them to preserve precious build fragments for the rest of time. Nor does it require some odd run-time remapping, or any additional mechanisms to centralise user management (e.g. LDAP. Please, no LDAP!) I think that tying ourselves and our operators into knots because we're afraid of the hit of one-time data migration, is crazy. AFAICS, the only risk left at that point, is elements that other people are maintaining. If we consider that to be a sufficient risk, we can still build the mechanism for injecting ID values from a previous build (essentially just seeding the static values that we'd be setting anyway) and apologise to the users who need that, or who don't discover its existence and break their clouds. Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from Chris Jones's message of 2014-10-07 04:35:33 -0700: Hi On 6 Oct 2014, at 17:41, Clint Byrum cl...@fewbar.com wrote: We have to be _extremely_ careful in how we manage this. I actually think it has potential to really blow up in our faces. Yes, anything we do here has the potential to be extremely ruinous for operators, but the reality is that any existing TripleO deployment is at pretty severe risk of blowing up because of UIDs/GIDs changing when they update. We need to give people a way to move forward without us merging a patch, and at the same time we need to make sure we provide a consistent set of UIDs for anything people may want to deploy with diskimage-builder. IMO the only desirable option *has* to be that we statically define UIDs and GIDs in the elements, because: 1: Requires no data fragments to be kept safe and fed to subsequent build processes 2: Doesn't do anything dynamic on first boot that could take hours/days 3: Can be thoroughly audited at build time to ensure correctness As you rightly point out though, any existing deployments will definitely be disrupted by this, but as I said above, all we'd be doing there is moving the needle from possible/probable to definite. Since the only leftovers we have from their previous image builds, are the images themselves, we could add the ability for a DIB run to extract IDs from a previous image, but this couldn't be required as a default build option, so we'd still risk existing deployments if they don't notice this feature. We could create a script that would spider an existing cloud and extract its ID mappings, to produce a fragment to feed into future builds, but again we're relying on operators to know that they need to do this. Wel... they'd know they need to do _something_ because their UIDs and GIDs are all horked up (technical term). Instead, I agree with Greg's view that this is our fault and we should fix it. We didn't think of this sooner, and as a result, our users are at risk. If we don't entirely fix this ourselves, we will be both expecting them to become aware of this issue and expecting them to do additional work to mitigate it. To that end, I think we should audit all of our elements for use of /mnt/state/ and use the specific knowledge we have of the software they relate to, to build one-time ID migration scripts, which would: 1: Execute before any related services start 2: Compare the now-static ID mappings against known files in /mnt/state 3: chown/chgrp any files/directories that need migrating 4: store a flag file in /mnt/state indicating that this process doesn't need to run again It does mean they have a potentially painfully long update process once, but the result will be a completely stable, static arrangement that will not require them to preserve precious build fragments for the rest of time. Nor does it require some odd run-time remapping, or any additional mechanisms to centralise user management (e.g. LDAP. Please, no LDAP!) I think that tying ourselves and our operators into knots because we're afraid of the hit of one-time data migration, is crazy. AFAICS, the only risk left at that point, is elements that other people are maintaining. If we consider that to be a sufficient risk, we can still build the mechanism for injecting ID values from a previous build (essentially just seeding the static values that we'd be setting anyway) and apologise to the users who need that, or who don't discover its existence and break their clouds. I'm not afraid of running migrations once. I want to make sure we never _plan_ to run migrations as part of regular operation. I agree with most of what you've written, but first I'd start with this: * Create an element which exports /etc/passwd and /etc/group from build process. * Create an element which imports /etc/passwd and /etc/group from local disk into image. This will have an element-provides of uid-gid-map * Create a separate element called 'static-users' which also provides uid-gid-map. Contains a map of uids and gids, and creates users early on with static UIDs/GIDs only. Disables usual commands used to add users and groups (error message should explain well enough that user can add their own element that provides uid-gid-map or switch to importing/exporting). * Make use-ephemeral depend on uid-gid-map. * Make tripleo-ci build with static-users, and recommend it in TripleO documentation. Once that is done, we will be producing builds with static users. If you want to create a user for base TripleO, you'll need to do it by hand in the static-users element. If you are downstream and want to do things differently that should be easy, just provide your own uid-gid-map element. As for migrations, that is fairly simple and can be done generically, I've already written a script that does it fairly reliably. The only worry is of course
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
On 8 October 2014 06:49, Clint Byrum cl...@fewbar.com wrote: Excerpts from Chris Jones's message of 2014-10-07 04:35:33 -0700: AFAICS, the only risk left at that point, is elements that other people are maintaining. If we consider that to be a sufficient risk, we can still build the mechanism for injecting ID values from a previous build (essentially just seeding the static values that we'd be setting anyway) and apologise to the users who need that, or who don't discover its existence and break their clouds. I'm not afraid of running migrations once. I want to make sure we never _plan_ to run migrations as part of regular operation. If I can make a suggestion... Set a marker in the image indicating whether it has fixed uuids or not. Calculate that by comparing the contents of the computed passwd and groups db's to the static reference we've put together. Secondly, at the end of migrations.d write a marker file in /mnt/state that indicates we're fully synced. This flag should contain a uuid of the image and whether or not that image had fixed ids - or something like that. Then, the chown migration should activate if either a) the image doesn't have fully fixed uuids, or b) the marker file is missing, or c) the marker file has a different uuid and indicates the prior image didn't have fixed ids. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Hi On 7 Oct 2014, at 18:49, Clint Byrum cl...@fewbar.com wrote: * Create an element which imports /etc/passwd and /etc/group from local disk into image. This will have an element-provides of uid-gid-map I don't think it should import those files wholesale, IMO that's making way too many assumptions about the similarity of the two builds. I think this needs to be at least somewhat driven from the elements themselves - they need to declare the users/groups they need and likely offer a default UID/GID for each. The element you describe could consult the elements to determine which users/groups need to be created, pull the values it needs from the passwd/group files if they exist, or fall back on the offered static defaults if not, then use normal tools to create the user/group (mainly so we respect whatever libnss settings the operator has chosen). * Create a separate element called 'static-users' which also provides uid-gid-map. Contains a map of uids and gids, and creates users early on I don't like the idea of keeping the map in a single element, for several reasons: 1: Harder to maintain across multiple element repos (e.g. we have two repos, do we have a static-users equivalent in each?) 2: Harder for downstreams to fork if they want to add an element we don't carry 3: Harder for devs to commit to, especially if it means they need to simultaneously land something in di-b and t-i-e. with static UIDs/GIDs only. Disables usual commands used to add users and groups (error message should explain well enough that user can add their Are you suggesting disabling those during build time only, or runtime too? I strongly dislike the latter and I'm not thrilled about the former. I'd rather we leave them as-is and audit the passwd/group files at the end of the build, vs what we were expecting from the elements. (we should also be aware that by enforcing this, we'll be increasing the number of elements we need to supply, because any dependencies that get pulled in during the build, which create users/groups, will now error/fail the build) As for migrations, that is fairly simple and can be done generically, I've already written a script that does it fairly reliably. The only Curious to see how that works - how can it know that /mnt/state/some/random/dir currently listed as ceilometer was actually owned by swift on the previous boot? Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from Chris Jones's message of 2014-10-07 12:23:20 -0700: Hi On 7 Oct 2014, at 18:49, Clint Byrum cl...@fewbar.com wrote: * Create an element which imports /etc/passwd and /etc/group from local disk into image. This will have an element-provides of uid-gid-map I don't think it should import those files wholesale, IMO that's making way too many assumptions about the similarity of the two builds. Disagree on the grounds that the point of this is to deal with people who already have images and want to update them. We're not trying to enable people to deploy radically different OS's or anything like that. I think this needs to be at least somewhat driven from the elements themselves - they need to declare the users/groups they need and likely offer a default UID/GID for each. The element you describe could consult the elements to determine which users/groups need to be created, pull the values it needs from the passwd/group files if they exist, or fall back on the offered static defaults if not, then use normal tools to create the user/group (mainly so we respect whatever libnss settings the operator has chosen). That is a lot of complexity for the case that we don't want people to use (constantly carrying /etc/passwd and /etc/group forward). * Create a separate element called 'static-users' which also provides uid-gid-map. Contains a map of uids and gids, and creates users early on I don't like the idea of keeping the map in a single element, for several reasons: 1: Harder to maintain across multiple element repos (e.g. we have two repos, do we have a static-users equivalent in each?) 2: Harder for downstreams to fork if they want to add an element we don't carry 3: Harder for devs to commit to, especially if it means they need to simultaneously land something in di-b and t-i-e. I think that makes sense. The per-element expressions will still need to be coalesced into one place to check for conflicts. I suppose if we use add only tools that will work. with static UIDs/GIDs only. Disables usual commands used to add users and groups (error message should explain well enough that user can add their Are you suggesting disabling those during build time only, or runtime too? I strongly dislike the latter and I'm not thrilled about the former. I'd rather we leave them as-is and audit the passwd/group files at the end of the build, vs what we were expecting from the elements. (we should also be aware that by enforcing this, we'll be increasing the number of elements we need to supply, because any dependencies that get pulled in during the build, which create users/groups, will now error/fail the build) So I think we still do care about these. MySQL puts its files on /mnt/state. RabbitMQ puts its files on /mnt/state. icinga puts its files on /mnt/state. If an element installs a package that needs a user, we have to treat that as our responsibility to handle, because we want to preserve /mnt/state. As for migrations, that is fairly simple and can be done generically, I've already written a script that does it fairly reliably. The only Curious to see how that works - how can it know that /mnt/state/some/random/dir currently listed as ceilometer was actually owned by swift on the previous boot? It also has to have access to the previous image's /etc/passwd and /etc/group. In the context I wrote it in, updating via ansible, I can cheat and copy that into /mnt/state before rebuilding the box. But we can do that in our elements too. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Hi On 7 Oct 2014, at 20:47, Clint Byrum cl...@fewbar.com wrote: I don't think it should import those files wholesale, IMO that's making way too many assumptions about the similarity of the two builds. Disagree on the grounds that the point of this is to deal with people who already have images and want to update them. We're not trying to enable people to deploy radically different OS's or anything like that. My point is that we don't know what else is changing under the hood. Switching OS is a bit extreme, but we don't know that they're not going to pull in an OS upgrade at the same time and have everything change substantially, or even just introduce one additional dependency which we would destroy the uid/gid for. I think this needs to be at least somewhat driven from the elements themselves - they need to declare the users/groups they need and likely offer a default UID/GID for each. The element you describe could consult the elements to determine which users/groups need to be created, pull the values it needs from the passwd/group files if they exist, or fall back on the offered static defaults if not, then use normal tools to create the user/group (mainly so we respect whatever libnss settings the operator has chosen). That is a lot of complexity for the case that we don't want people to use (constantly carrying /etc/passwd and /etc/group forward). As tchaypo pointed out on IRC, if we do this static route, we are laying down a great big concrete slab of opinion in our images. I'm all for opinionated software, but we need to give people an out, which means we probably want to have a way for the suggested default UID/GIDs to be overridden, i.e. roughly what I described. We can just use that override to inject the pre-existing passwd/group values, if we so desire. I think that makes sense. The per-element expressions will still need to be coalesced into one place to check for conflicts. I suppose if we Definitely. If we use normal system tools to create the users/groups, the build will fail if there are conflicts. So I think we still do care about these. MySQL puts its files on /mnt/state. RabbitMQ puts its files on /mnt/state. icinga puts its files on /mnt/state. If an element installs a package that needs a user, we have to treat that as our responsibility to handle, because we want to preserve /mnt/state. Agreed. I just wanted to put a note on the table that we will have to care about these things :) It also has to have access to the previous image's /etc/passwd and /etc/group. In the context I wrote it in, updating via ansible, I can *nods* Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from Chris Jones's message of 2014-10-07 13:27:07 -0700: Hi On 7 Oct 2014, at 20:47, Clint Byrum cl...@fewbar.com wrote: I don't think it should import those files wholesale, IMO that's making way too many assumptions about the similarity of the two builds. Disagree on the grounds that the point of this is to deal with people who already have images and want to update them. We're not trying to enable people to deploy radically different OS's or anything like that. My point is that we don't know what else is changing under the hood. Switching OS is a bit extreme, but we don't know that they're not going to pull in an OS upgrade at the same time and have everything change substantially, or even just introduce one additional dependency which we would destroy the uid/gid for. Ok, any time we can fail at build time is good, so merging is important. I agree now. I think this needs to be at least somewhat driven from the elements themselves - they need to declare the users/groups they need and likely offer a default UID/GID for each. The element you describe could consult the elements to determine which users/groups need to be created, pull the values it needs from the passwd/group files if they exist, or fall back on the offered static defaults if not, then use normal tools to create the user/group (mainly so we respect whatever libnss settings the operator has chosen). That is a lot of complexity for the case that we don't want people to use (constantly carrying /etc/passwd and /etc/group forward). As tchaypo pointed out on IRC, if we do this static route, we are laying down a great big concrete slab of opinion in our images. I'm all for opinionated software, but we need to give people an out, which means we probably want to have a way for the suggested default UID/GIDs to be overridden, i.e. roughly what I described. We can just use that override to inject the pre-existing passwd/group values, if we so desire. Mmk. I think having an override is important, but I feel that we can do it in a more straight forward manner. I may not have thought it all the way through. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from Clint Byrum's message of 2014-10-02 14:15:30 +: Excerpts from Gregory Haynes's message of 2014-10-01 19:09:38 -0700: If we really want to go with this type of aproach we could also just copy the existing /etc/passwd into the image thats being built. Then when users are added they should be added in after existing users. I do like this approach, and it isn't one I had considered. We will know what image we want to update from in nearly every situation. Also this supports another case, which is rolling back to the previous image, quite well. Really this is just an automated form of static UID assignment. Now that ive proposed this id like to make an argument against the copy /etc/passwd as our long term solution (sorry). I do think its a not bad immediate fix, but long term id prefer actual static UID assignment out of the solutions proposed so far. It seems like determining how to build a new image based on the state of a previous image is an exact anti-pattern that read only / ephemeral instances aim to solve - minimize entropy collected over time from doing updates. Were also adding a requirement that user databases are now precious data which cannot ever be lost for a given image type. Its worth noting that these are both issues that operators will encounter. Static UID/GID assignment requires more developer work but AFAICT it passes less potential issues off onto operators (which should be our goal). Cheers, Greg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from Gregory Haynes's message of 2014-10-06 07:06:02 -0700: Excerpts from Clint Byrum's message of 2014-10-02 14:15:30 +: Excerpts from Gregory Haynes's message of 2014-10-01 19:09:38 -0700: If we really want to go with this type of aproach we could also just copy the existing /etc/passwd into the image thats being built. Then when users are added they should be added in after existing users. I do like this approach, and it isn't one I had considered. We will know what image we want to update from in nearly every situation. Also this supports another case, which is rolling back to the previous image, quite well. Really this is just an automated form of static UID assignment. Now that ive proposed this id like to make an argument against the copy /etc/passwd as our long term solution (sorry). I do think its a not bad immediate fix, but long term id prefer actual static UID assignment out of the solutions proposed so far. We have to be _extremely_ careful in how we manage this. I actually think it has potential to really blow up in our faces. We need to give people a way to move forward without us merging a patch, and at the same time we need to make sure we provide a consistent set of UIDs for anything people may want to deploy with diskimage-builder. It seems like determining how to build a new image based on the state of a previous image is an exact anti-pattern that read only / ephemeral instances aim to solve - minimize entropy collected over time from doing updates. Were also adding a requirement that user databases are now precious data which cannot ever be lost for a given image type. Its worth noting that these are both issues that operators will encounter. I disagree with some of these points. Expecting Linux filesystems to be consumable from any machine without a source of consistent UIDs/GIDs is the anti-pattern. Using /etc/passwd and /etc/group from the previous image is just one such source, and it is one source that anybody who has a previous image will always have. If you've deployed using TripleO's tools already, you will not have the same UIDs that our centralized UID registry would have. We could also just deploy FreeIPA on all controllers, delay UID allocations, and put everything in LDAP. Having our own special project for static UIDs feels like a third possible implementation, not one I'm ready to say is the way. I'm leaning more toward keeping a copy of the UID/GID/symbolic-name database anywhere that has state, and use a guard against making use of a system database which conflicts with existing mappings on the state drive. Then we can use either a static UID project, or previous images, as the source of UIDs/GIDs for new images, to avoid hitting the guard. Static UID/GID assignment requires more developer work but AFAICT it passes less potential issues off onto operators (which should be our goal). +1 for making this as painless as possible on operators. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from James Polley's message of 2014-10-02 05:37:25 +: All three of the options presented here seem to assume that UIDs will always be allocated at image-build time. I think that's because most of these UIDs will be used to write files into the chroot at image-create time - if I could think of some way around that, I think we could avoid this problem more neatly by not assigning the UIDs until first boot But since we can't do that, would it be possible to compromise by having the UIDs read in from heat metadata, and using the current allocation process if none is provided? This should allow people who prefer to have static UIDs to have simple drop-in config, but also allow people who want to dynamically read from existing images to scrape the details and then drop them in. To aid people who have existing images, perhaps we could provide a small tool (if one doesn't already exist) that simply reads /etc/passwd and returns a JSON username:uid map, to be added into the heat local environment when building the next image? What I was suggesting before as an alternate solution is a more simple version of this - just copy the existing /etc/passwd and friends into the chroot at the start of building a new image. This should cause new users to be created in a safe way. IMO I like the uid pinning better as a solution, though. Cheers, Greg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
-Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 02 October 2014 02:51 To: openstack-dev Subject: [openstack-dev] [TripleO] a need to assert user ownership in preserved state Recently we've been testing image based updates using TripleO, and we've run into an interesting conundrum. Currently, our image build scripts create a user per service for the image. We don't, at this time, assert a UID, so it could get any UID in the /etc/passwd database of the image. However, if we add a service that happens to have its users created before a previously existing service, the UID's shift by one. When this new image is deployed, the username might be 'ceilometer', but /mnt/state/var/lib/ceilometer is now owned by 'cinder'. Here are 3 approaches, which are not mutually exclusive to one another. There are likely others, and I'd be interested in hearing your ideas. * Static UID's for all state-preserving services. Basically we'd just allocate these UID's from a static pool and those are always the UIDs no matter what. This is the simplest solution, but does not help anybody who is already looking to update a TripleO cloud. Also, this would cause problems if TripleO wants to merge with any existing system that might also want to use similar UID's. This also provides no guard against non-static UID's storing things on the state partition. * Fix the UID's on image update. We can backup /etc/passwd and /etc/group to /mnt/state, and on bootup we can diff the two, and any UIDs that changed can be migrated. This could be very costly if the swift storage UID changed, with millions of files present on the system. This merge process is also not atomic and may not be reversible, so it is a bit scary to automate this. * Assert ownership when registering state path. We could have any state-preserving elements register their desire for any important globs for the state drive to be owned by a particular symbolic username. This is just a different, more manual way to fix the UID's and carries the same cons. For these last two cases, of fixing the file ownership on first boot based on the previous UIDs of a username, why would we decide to fix the data files? If instead we were to change the UIDs such that the data files were correct, the only thing to fix up would be the installed files in the image, which are a well-defined and limited set of files. So, what do people think? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Thanks, Jon-Paul Sullivan ☺ Cloud Services - @hpcloud Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, Galway. Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's Quay, Dublin 2. Registered Number: 361933 The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as HP CONFIDENTIAL. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from Gregory Haynes's message of 2014-10-01 19:09:38 -0700: Excerpts from Clint Byrum's message of 2014-10-02 01:50:33 +: Recently we've been testing image based updates using TripleO, and we've run into an interesting conundrum. Currently, our image build scripts create a user per service for the image. We don't, at this time, assert a UID, so it could get any UID in the /etc/passwd database of the image. However, if we add a service that happens to have its users created before a previously existing service, the UID's shift by one. When this new image is deployed, the username might be 'ceilometer', but /mnt/state/var/lib/ceilometer is now owned by 'cinder'. Wow, nice find! Indeed, the Helion dev team discovered this one whilst working on updating between alternating builds that added or removed some services. Oh, I forgot to mention the bug reference: https://bugs.launchpad.net/tripleo/+bug/1374626 Here are 3 approaches, which are not mutually exclusive to one another. There are likely others, and I'd be interested in hearing your ideas. * Static UID's for all state-preserving services. Basically we'd just allocate these UID's from a static pool and those are always the UIDs no matter what. This is the simplest solution, but does not help anybody who is already looking to update a TripleO cloud. Also, this would cause problems if TripleO wants to merge with any existing system that might also want to use similar UID's. This also provides no guard against non-static UID's storing things on the state partition. +1 for this approach for the reasons mentioned. * Fix the UID's on image update. We can backup /etc/passwd and /etc/group to /mnt/state, and on bootup we can diff the two, and any UIDs that changed can be migrated. This could be very costly if the swift storage UID changed, with millions of files present on the system. This merge process is also not atomic and may not be reversible, so it is a bit scary to automate this. If we really want to go with this type of aproach we could also just copy the existing /etc/passwd into the image thats being built. Then when users are added they should be added in after existing users. I do like this approach, and it isn't one I had considered. We will know what image we want to update from in nearly every situation. Also this supports another case, which is rolling back to the previous image, quite well. Really this is just an automated form of static UID assignment. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from James Polley's message of 2014-10-01 22:37:25 -0700: All three of the options presented here seem to assume that UIDs will always be allocated at image-build time. I think that's because most of these UIDs will be used to write files into the chroot at image-create time - if I could think of some way around that, I think we could avoid this problem more neatly by not assigning the UIDs until first boot Yeah I don't think we're going to work around that. It is part of the magic of images that the metadata is all in place and there's no churn at boot. But since we can't do that, would it be possible to compromise by having the UIDs read in from heat metadata, and using the current allocation process if none is provided? I really, really dislike this. Post-boot tools like Heat are for per-server customization and site-wide changes. UIDs seem like plumbing under the hood. This should allow people who prefer to have static UIDs to have simple drop-in config, but also allow people who want to dynamically read from existing images to scrape the details and then drop them in. I see your point, and I'm now confused as I don't really understand what would make somebody prefer dynamic UID allocation. To aid people who have existing images, perhaps we could provide a small tool (if one doesn't already exist) that simply reads /etc/passwd and returns a JSON username:uid map, to be added into the heat local environment when building the next image? Or a tool that reads the image, and returns /etc/passwd and /etc/group. Thanks very much for your thoughts. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
-Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 02 October 2014 15:16 To: openstack-dev Subject: Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state Excerpts from Gregory Haynes's message of 2014-10-01 19:09:38 -0700: Excerpts from Clint Byrum's message of 2014-10-02 01:50:33 +: Recently we've been testing image based updates using TripleO, and we've run into an interesting conundrum. Currently, our image build scripts create a user per service for the image. We don't, at this time, assert a UID, so it could get any UID in the /etc/passwd database of the image. However, if we add a service that happens to have its users created before a previously existing service, the UID's shift by one. When this new image is deployed, the username might be 'ceilometer', but /mnt/state/var/lib/ceilometer is now owned by 'cinder'. Wow, nice find! Indeed, the Helion dev team discovered this one whilst working on updating between alternating builds that added or removed some services. Oh, I forgot to mention the bug reference: https://bugs.launchpad.net/tripleo/+bug/1374626 Here are 3 approaches, which are not mutually exclusive to one another. There are likely others, and I'd be interested in hearing your ideas. * Static UID's for all state-preserving services. Basically we'd just allocate these UID's from a static pool and those are always the UIDs no matter what. This is the simplest solution, but does not help anybody who is already looking to update a TripleO cloud. Also, this would cause problems if TripleO wants to merge with any existing system that might also want to use similar UID's. This also provides no guard against non-static UID's storing things on the state partition. +1 for this approach for the reasons mentioned. * Fix the UID's on image update. We can backup /etc/passwd and /etc/group to /mnt/state, and on bootup we can diff the two, and any UIDs that changed can be migrated. This could be very costly if the swift storage UID changed, with millions of files present on the system. This merge process is also not atomic and may not be reversible, so it is a bit scary to automate this. If we really want to go with this type of aproach we could also just copy the existing /etc/passwd into the image thats being built. Then when users are added they should be added in after existing users. I do like this approach, and it isn't one I had considered. We will know what image we want to update from in nearly every situation. Also this supports another case, which is rolling back to the previous image, quite well. Really this is just an automated form of static UID assignment. So for situations where images are built and then distributed 2 things would need to happen: 1. Identify new users and add them to the passswd file. 2. Modify ownership of files in the new image whose owners UID have changed, as per my previous mail. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Thanks, Jon-Paul Sullivan ☺ Cloud Services - @hpcloud Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, Galway. Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's Quay, Dublin 2. Registered Number: 361933 The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as HP CONFIDENTIAL. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from Sullivan, Jon Paul's message of 2014-10-02 02:08:26 -0700: -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 02 October 2014 02:51 To: openstack-dev Subject: [openstack-dev] [TripleO] a need to assert user ownership in preserved state Recently we've been testing image based updates using TripleO, and we've run into an interesting conundrum. Currently, our image build scripts create a user per service for the image. We don't, at this time, assert a UID, so it could get any UID in the /etc/passwd database of the image. However, if we add a service that happens to have its users created before a previously existing service, the UID's shift by one. When this new image is deployed, the username might be 'ceilometer', but /mnt/state/var/lib/ceilometer is now owned by 'cinder'. Here are 3 approaches, which are not mutually exclusive to one another. There are likely others, and I'd be interested in hearing your ideas. * Static UID's for all state-preserving services. Basically we'd just allocate these UID's from a static pool and those are always the UIDs no matter what. This is the simplest solution, but does not help anybody who is already looking to update a TripleO cloud. Also, this would cause problems if TripleO wants to merge with any existing system that might also want to use similar UID's. This also provides no guard against non-static UID's storing things on the state partition. * Fix the UID's on image update. We can backup /etc/passwd and /etc/group to /mnt/state, and on bootup we can diff the two, and any UIDs that changed can be migrated. This could be very costly if the swift storage UID changed, with millions of files present on the system. This merge process is also not atomic and may not be reversible, so it is a bit scary to automate this. * Assert ownership when registering state path. We could have any state-preserving elements register their desire for any important globs for the state drive to be owned by a particular symbolic username. This is just a different, more manual way to fix the UID's and carries the same cons. For these last two cases, of fixing the file ownership on first boot based on the previous UIDs of a username, why would we decide to fix the data files? If instead we were to change the UIDs such that the data files were correct, the only thing to fix up would be the installed files in the image, which are a well-defined and limited set of files. Great point JP! I think that this is similar to Greg's suggestion of copying the uid/gid database into the image. If we copy it in before the image is built, we fix the image by building it right in the first place. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
On 3 Oct 2014, at 00:25, Clint Byrum cl...@fewbar.com wrote: Excerpts from James Polley's message of 2014-10-01 22:37:25 -0700: All three of the options presented here seem to assume that UIDs will always be allocated at image-build time. I think that's because most of these UIDs will be used to write files into the chroot at image-create time - if I could think of some way around that, I think we could avoid this problem more neatly by not assigning the UIDs until first boot Yeah I don't think we're going to work around that. It is part of the magic of images that the metadata is all in place and there's no churn at boot. Agree - it would be quite a significant change in how TripleO works, not just a workaround. But since we can't do that, would it be possible to compromise by having the UIDs read in from heat metadata, and using the current allocation process if none is provided? I really, really dislike this. Post-boot tools like Heat are for per-server customization and site-wide changes. UIDs seem like plumbing under the hood. I think that the part of this you dislike is specifically storing the data in heat? Would you object less if I phrased it as a job file to be read at image build time, which is closer to what I had in mind? This should allow people who prefer to have static UIDs to have simple drop-in config, but also allow people who want to dynamically read from existing images to scrape the details and then drop them in. I see your point, and I'm now confused as I don't really understand what would make somebody prefer dynamic UID allocation. I was thinking of a case where an operator might have several existing images with different sets of services, or different base distribtions, and hence different sets of uids; they'd probably prefer to have the build process extract the details from the previous image rather than having a single fixed map of uids. Someone starting fresh might prefer to provide a static map of pre-assigned UIDs To aid people who have existing images, perhaps we could provide a small tool (if one doesn't already exist) that simply reads /etc/passwd and returns a JSON username:uid map, to be added into the heat local environment when building the next image? Or a tool that reads the image, and returns /etc/passwd and /etc/group. Sure, but I think it would be handy if it could accept data from another source as well as the previous image, to cater for people who want to be more prescriptive about which UIDs are used but don't have adv Bing existing image yet. I don't know if this is a real use case though - maybe I'm just remembering bad experiences from a previous pre-cloud life. Thanks very much for your thoughts. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
On 3 Oct 2014, at 02:57, James Polley j...@jamezpolley.com wrote: On 3 Oct 2014, at 00:25, Clint Byrum cl...@fewbar.com wrote: Excerpts from James Polley's message of 2014-10-01 22:37:25 -0700: All three of the options presented here seem to assume that UIDs will always be allocated at image-build time. I think that's because most of these UIDs will be used to write files into the chroot at image-create time - if I could think of some way around that, I think we could avoid this problem more neatly by not assigning the UIDs until first boot Yeah I don't think we're going to work around that. It is part of the magic of images that the metadata is all in place and there's no churn at boot. Agree - it would be quite a significant change in how TripleO works, not just a workaround. But since we can't do that, would it be possible to compromise by having the UIDs read in from heat metadata, and using the current allocation process if none is provided? I really, really dislike this. Post-boot tools like Heat are for per-server customization and site-wide changes. UIDs seem like plumbing under the hood. I think that the part of this you dislike is specifically storing the data in heat? Would you object less if I phrased it as a job file to be read at image build time, which is closer to what I had in mind? This should allow people who prefer to have static UIDs to have simple drop-in config, but also allow people who want to dynamically read from existing images to scrape the details and then drop them in. I see your point, and I'm now confused as I don't really understand what would make somebody prefer dynamic UID allocation. I was thinking of a case where an operator might have several existing images with different sets of services, or different base distribtions, and hence different sets of uids; they'd probably prefer to have the build process extract the details from the previous image rather than having a single fixed map of uids. Someone starting fresh might prefer to provide a static map of pre-assigned UIDs To be clear - I don't think either of these is novel - these are cases 1 and 2 from the mail that started the thread. The point I'm ineptly trying to make (why am I sending email at 3am?) is that I think we can easily support both 1 and 2 simply by thinking of read list of UIDs from an existing image and apply existing list of UIDs to new image as separate tasks and implement both separately To aid people who have existing images, perhaps we could provide a small tool (if one doesn't already exist) that simply reads /etc/passwd and returns a JSON username:uid map, to be added into the heat local environment when building the next image? Or a tool that reads the image, and returns /etc/passwd and /etc/group. Sure, but I think it would be handy if it could accept data from another source as well as the previous image, to cater for people who want to be more prescriptive about which UIDs are used but don't have adv Bing existing image yet. I don't know if this is a real use case though - maybe I'm just remembering bad experiences from a previous pre-cloud life. Thanks very much for your thoughts. :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Excerpts from Clint Byrum's message of 2014-10-02 01:50:33 +: Recently we've been testing image based updates using TripleO, and we've run into an interesting conundrum. Currently, our image build scripts create a user per service for the image. We don't, at this time, assert a UID, so it could get any UID in the /etc/passwd database of the image. However, if we add a service that happens to have its users created before a previously existing service, the UID's shift by one. When this new image is deployed, the username might be 'ceilometer', but /mnt/state/var/lib/ceilometer is now owned by 'cinder'. Wow, nice find! Here are 3 approaches, which are not mutually exclusive to one another. There are likely others, and I'd be interested in hearing your ideas. * Static UID's for all state-preserving services. Basically we'd just allocate these UID's from a static pool and those are always the UIDs no matter what. This is the simplest solution, but does not help anybody who is already looking to update a TripleO cloud. Also, this would cause problems if TripleO wants to merge with any existing system that might also want to use similar UID's. This also provides no guard against non-static UID's storing things on the state partition. +1 for this approach for the reasons mentioned. * Fix the UID's on image update. We can backup /etc/passwd and /etc/group to /mnt/state, and on bootup we can diff the two, and any UIDs that changed can be migrated. This could be very costly if the swift storage UID changed, with millions of files present on the system. This merge process is also not atomic and may not be reversible, so it is a bit scary to automate this. If we really want to go with this type of aproach we could also just copy the existing /etc/passwd into the image thats being built. Then when users are added they should be added in after existing users. I still prefer the first solution, though. * Assert ownership when registering state path. We could have any state-preserving elements register their desire for any important globs for the state drive to be owned by a particular symbolic username. This is just a different, more manual way to fix the UID's and carries the same cons. So, what do people think? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Clint Byrum cl...@fewbar.com wrote on 10/01/2014 09:50:33 PM: Recently we've been testing image based updates using TripleO, and we've run into an interesting conundrum. Currently, our image build scripts create a user per service for the image. We don't, at this time, assert a UID, so it could get any UID in the /etc/passwd database of the image. However, if we add a service that happens to have its users created before a previously existing service, the UID's shift by one. When this new image is deployed, the username might be 'ceilometer', but /mnt/state/var/lib/ceilometer is now owned by 'cinder'. I do not understand the problem statement. Unfortunately, I am not familiar with image based updates using TripleO. What is updating what? If the UIDs are not asserted, what UIDs shift by one? Is this because some files keep owner UID while the some UID=name binding in /etc/passwd changes? Or the other way around? Why would there be a change in either? If there is no short answer, don't worry about it. Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
On 02/10/14 12:26, Mike Spreitzer wrote: I do not understand the problem statement. Unfortunately, I am not familiar with image based updates using TripleO. What is updating what? If the UIDs are not asserted, what UIDs shift by one? Is this because some files keep owner UID while the some UID=name binding in /etc/passwd changes? Or the other way around? Why would there be a change in either? If there is no short answer, don't worry about it. You build one image, which creates a 'ceilometer' user with UID 1001, and deploy that to your cloud. You then build a new image to deploy to your cloud, and due to element ordering (or something else), the 'cinder' user now has UID 1001. When you switch to this image, /mnt/state/var/lib/ceilometer is now owned by cinder. Cheers, -- Steve If it (dieting) was like a real time strategy game, I'd have loaded a save game from ten years ago. - Greg, Columbia Internet ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
All three of the options presented here seem to assume that UIDs will always be allocated at image-build time. I think that's because most of these UIDs will be used to write files into the chroot at image-create time - if I could think of some way around that, I think we could avoid this problem more neatly by not assigning the UIDs until first boot But since we can't do that, would it be possible to compromise by having the UIDs read in from heat metadata, and using the current allocation process if none is provided? This should allow people who prefer to have static UIDs to have simple drop-in config, but also allow people who want to dynamically read from existing images to scrape the details and then drop them in. To aid people who have existing images, perhaps we could provide a small tool (if one doesn't already exist) that simply reads /etc/passwd and returns a JSON username:uid map, to be added into the heat local environment when building the next image? On 2 Oct 2014, at 11:50, Clint Byrum cl...@fewbar.com wrote: Recently we've been testing image based updates using TripleO, and we've run into an interesting conundrum. Currently, our image build scripts create a user per service for the image. We don't, at this time, assert a UID, so it could get any UID in the /etc/passwd database of the image. However, if we add a service that happens to have its users created before a previously existing service, the UID's shift by one. When this new image is deployed, the username might be 'ceilometer', but /mnt/state/var/lib/ceilometer is now owned by 'cinder'. Here are 3 approaches, which are not mutually exclusive to one another. There are likely others, and I'd be interested in hearing your ideas. * Static UID's for all state-preserving services. Basically we'd just allocate these UID's from a static pool and those are always the UIDs no matter what. This is the simplest solution, but does not help anybody who is already looking to update a TripleO cloud. Also, this would cause problems if TripleO wants to merge with any existing system that might also want to use similar UID's. This also provides no guard against non-static UID's storing things on the state partition. * Fix the UID's on image update. We can backup /etc/passwd and /etc/group to /mnt/state, and on bootup we can diff the two, and any UIDs that changed can be migrated. This could be very costly if the swift storage UID changed, with millions of files present on the system. This merge process is also not atomic and may not be reversible, so it is a bit scary to automate this. * Assert ownership when registering state path. We could have any state-preserving elements register their desire for any important globs for the state drive to be owned by a particular symbolic username. This is just a different, more manual way to fix the UID's and carries the same cons. So, what do people think? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
