hey shawn,
thanks for looking this over. i've addressed and deleted all the s///
style comments below and my replies to the other issues are inline.
ed
On Mon, May 24, 2010 at 01:39:43PM -0700, Shawn Walker wrote:
> On 03/ 7/10 12:23 PM, Edward Pilatowicz wrote:
> >hey all,
> >
> >so in an effort to get pkg image-update to work with zones, i've been
> >prototyping support for linked images within pkg(5). i don't have a fully
> >functional prototype to pass around yet, but i have documented some
> >aspects of my prototype in an initial design doc. i've attached that doc
> >so that people can take a look, see what i'm thinking, provide feed
> >back, etc. all comments and criticism are welcome.
>
> lines 10-34:
> So are linked image types just *additional* types of images (e.g.
> full, partial, etc.) or is this a subtype of an existing image type?
>
hm. so i didn't even realize that pkg had support like things for
partial and user images... looking at the code i see that
IMG_TYPE_PARTIAL and IMG_TYPE_USER are not implemented. so lucky for me
i don't have a lot of re-designing todo to integrate with these types
of images. ;)
my current prototype doesn't really match either description you've
given. i don't have any new IMG_TYPE_* types, and i also am not
creating any new classes derived from the Image class. so far, i've
been think of linking as more of an attribute of two images, and the
link type is then an attribute of the linking.
> lines 164-205: User linked images:
> I'd like to make the observation that it seems like user and zone
> linked images fall into two high level models:
>
> - push type
>
> This is roughly what you describe zone images as; a push-only model
> where the parent tells any child images what the new state of the
> parent image is after operations are completed.
>
> - pull type
>
> This is what you describe user images as currently; a pull-only
> model where the user or some service is responsible for detecting
> changes in a parent image and syncing the child user image.
>
> I'd suggest that user images could be both push and pull. That is,
> an administrator may find it useful to create a special set of
> user images for creating isolated software stacks of specific
> packages that can't normally co-exist. This is often seen with
> software installed in /opt as an example.
>
correct. but user images are out of scope for the current project. i
talk about them as an example of what else is possible.
but note that user linked images are really an extension of default
linked images. (which are in scope for the project.) default linked
images just assume a push model and a user of root.
so i've been using the terminology export/import where you're saying
push/pull. would you like me to change the terminology i'm currently
using?
> lines 253ff:
> Indeed, instead of "system publisher", I'd prefer to see the text
> "system repository" or "parent image repository" since that's what
> the functionality really is.
>
i've renamed this and updated this section some.
> lines 270-288:
> I'm assuming that this complements / assumes implementation of
> bug 15343?
>
i've added a mention of 15343
> lines 320-330:
> It seems like it would be helpful if beadm permitted linking any
> arbitrary ZFS filesystem to a given BE to enable this functionality.
> In particular, I was thinking that it would be useful for
> administrators to choose whether some filesystems remained
> independent or stayed in sync with a given boot environment.
> This could be particularly useful in the linked user image
> management of isolated software stacks I mentioned above or
> for the management of configuration data.
>
> I agree that this enhancement is out of scope for this proposal,
> but wanted to mention it.
>
agreed.
> lines 333-340:
> I could be mistaken, but I seem to recall from my past experience
> with translation and localisation that the terms parent/child
> were strongly preferred in general end-user documentation over
> master/slave. I'd ask someone in l10n about this and then only
> use the terms they recommend.
>
i'm not wedded in any way to the master/slave terminology.
i've sent an email to our docs person asking for input.
> lines 375-384:
> When you say "constraints data", you really mean "image state",
> right? Or you imply "constraints data" in the more generic sense
> (e.g. package versions, package state, package constraints)?
>
to dive into details, the "constraints data" maps to:
- a list of packages (which includes their full version numbers and
publishers) installed in the parent image. this list doesn't include
ALL the packages in the parent image, just those required to implement
the specified image syncing policy. this data could arguably be called
the parent data "image state", but it's not the complete state.
- some linked image property values. , linked image properties may have
different authoritative sources for different types of linked images.
for example, for zones the pkg synchronization policy will be coming
from the global zone. so in this case we need to tell the zone what
to keep in sync. while for user images, the image itself can decide
what it wants to keep in sync so it doesn't need to have that property
exported to it. this doesn't really seem like parent data "image
state".
> lines 386-391:
> I think /var/pkg/state/export /var/pkg/state/import or
> /var/pkg/cache/state/export, /var/pkg/cache/state/import
> would fit better with the on-disk format proposal. I don't
> understand why 'master' is part of the pathname here since
> there's no corresponding linked/child/* set of directories
> in this proposal either.
>
master is there because i've gone through lots of iterations, some of
which used to have a "child" directory. my current prototype doesn't
have "child" directories, but it might have to come back.
the underlying problem is that the gz can't really safely write directly
to ngz filesystems. this makes exporting data to a running zone
difficult. in my current prototype i use fork() and chroot() to solve
this problem. another way to solve this problem would be to have
"child" directories that the master image that the gz writes into, and
those directories are read only lofs mounted into each zone before that
zone is booted.
wrt location, i have no issues with moving the directories around to
/var/pkg/state or /var/pkg/cache. but i do like having "linked" in the
path name designate that this as linked image data.
> lines 393-399:
> This bit is really confusing to me:
>
> - What does the linked image type and content policy stored here
> indicate?
>
um. it indicates the type of linked image and the constraints on it's
content?
so for example. in a zone the type would be zone. in a default image
the type would be default.
for the content policy, it would be whatever the user had configured.
in zones, the minimum content policy value would be to keep cip packages
in sync. (which will just be called minimum.) there could be other
content policies that would perhaps say "keep all packages in sync".
> - How does that meaning change based on whether it is in the export
> or import directory?
>
the meaning doesn't change. the import vs export directories are more
about who can write to the directories and where the data is coming
from. as mentioned previously, in some cases i though that the export
directory might actually be a read-only lofs mount.
> - What form is this information stored in and when does it get
> stored?
>
the data is stored in private .xml files.
exported data gets written:
- an operation on the parent image changes the parent image state. for
example, a pkg install/uninstall/image-update on the parent image.
- an explicit "pkg linked-export" is done.
- an linked image operation initiated from the parent image. for
example, if you do "pkg linked-sync -l <linked-image-name>" then we'll
re-export the linked image data before starring the linked-sync
operation on the specified image.
i haven't implemented the import/pull yet so i can't really nail down
when that will occur, but my best guess is that any pkg operation on an
child image which is going to write to the image would take a write lock
on the image, update the import/pull data, and then starting it's
operation.
> - It seems like images that you *push* changes to should simply get
> their data from a temporary directory that contains the exported
> information for the duration of the operation. I'm not sure this
the exported/pushed constraint data really needs to persist across
operations. the reason for this is that not all linked image operations
are initiated from the parent image. for example, someone within the
zone might run pkg(1) to install some new software, and that command
should not allow installation of software which would violate the
constraints on the image. i tried to call this out in the "zones
requirements" section:
- all the constraints required to perform software management operations
within a non-global zone must be available within that zone.
> should be stored in $IMGROOT/pkg. For images that *pull* changes
> from the master image, realistically this information needs to be
> generated on demand instead of being stored in $IMGROOT/pkg. If
> the purpose of this directory is to simply cache information that
> was imported or is being used to perform a sync operation, then
> I'd say it should live under /var/pkg/cache/linked (to fit with
> the on-disk proposal). Does this data need to be kept longer
> than that?
>
for pull image this is debatable. i was imaging that there would be a
desire to be able to do operations on linked child images when the
parent image is not accessible. for example, if i had a user image in
my home directory, which is nfs accessible from any machine, and that
image was currently sync'd up with jurassic, should i be able to install
packages into that image on another machine and still have the linked
image constraints apply to that operation? if so the information needs
to be persisted within the image.
> - I think that if you're exporting "constraint information" that it
> really sounds like you should just be generating an incorporation
> package since that's equivalent and fits with existing
> functionality. In particular, I would anticipate pkg freeze
> to be based on incorporations so that would fit better with what
> you've proposed here.
>
well, kinda. i certainly could represent what's installed in the
gz/parent as a package that use "require" and "incorporate"
dependencies. this is actually, what i do, just all in memory. (when
pkg runs it reads the constraint information from disk and generates
packages based on it.
now i'm going to guess that your next suggestion will be to do this but
just on disk. ie, write the constraint package into the child image and
into the child images installed catalog. unfortunately i don't really
want to do this, and there are a few reasons.
- this requires that the parent image constantly update lots of data in
the client image. and as i've already mentioned, in the case of zones
this represents a security issue. we can't trust any data in the
child images, so reading in catalogs is a problem unless we've audited
all the python code involved with parsing catalogs. i'd really prefer
to avoid that.
- also, with this approach there is no good way to deal with out of sync
images. ie, images that don't adhere to their sync policy. you can't
represent out-of-sync images in the installed catalog because you'd
have an invalid state, which leads to a very unhappy sat solver.
> - If you're going to export state information for the parent image,
> then that's what's contained in the /var/pkg/state/installed
> directory. Specifically, just catalog.attrs and catalog.base.C.
>
i'm aware of the data stored in there. but i don't actually want to
export all that data. a zone should not know more about the global zone
than it needs to. the ngz needs to know about the software installed in
the gz that needs to be keep in sync with the gz, but it doesn't
necessarily need to know about all the monitoring and management
software installed in the gz.
so really the zone needs to know about a subset of the information in
catalog.attrs and catalog.base.C. now admittingly, i could represent
that data in the same format as the current catalog.attrs and
catalog.base.C files, but i haven't seen any advantage to doing that.
as i mentioned before, currently i just have an xml file that lists the
information, but i'm not wedded to that, it's more an implementation
detail.
> lines 401-407:
> If we're going to go this route, I'd like to see a documented
> serialisation format for the imageplan that can be used more
> generically. It should also be JSON; not pickle-based. You
> need a more general format anyway to fit with some of the
> functionality described later on in this proposal for --runid.
>
ok. so i'm hoping to nuke the --runid and --plan stuff. i'm currently
planning to replace both with a simple control pipe that will manage the
stages of the pkg planning operations in sync across the images.
also, the way i'm representing this data is in the same format as the
constrained packages. ie. once again i just have a list of packages
that will be installed after the operation complete that are affected by
the sync policy. i'm not actually exporting the entire plan.
> lines 421-427:
> Since a linked image can realistically only account for the last
> sync'd state of a parent image, it seems like the parent image
> could simply provide constraints as a dynamically generated
> incorporation package (manifest) as part of the export/import
> process. That manifest could then be stored locally and treated
> exactly like a package normally would be without any special logic.
> Doing so also makes it possible for the normal memory management
> that the client api uses internally to not have to have special
> logic to marshal this information to disk (possibly repeatedly).
>
possibly, but isn't manifest dependency information cached in the
catalog? if so this would require re-writing the catalog in zones which
requires reading the catalog in zones. as i've pointed out before, that
would be bad.
in case it isn't obvious, i'm really trying hard keep data flowing in
one direction. from the parent image to the child. i'm also trying to
keep the data flow simple. having pkg traverse directories in a zone
image and read data from that zone isn't safe. i tried to call this out
early in the document with in the "zones requirements" section:
- since zones are untrusted execution environment, global zone pkg(5)
operations should not be required to read data from non-global zone.
ie, any data flow required to support linked images must be from a
global zone to a non-global zone, and never in the reverse direction.
also, there is one more thing to consider. not all the linked image
constraints on an image can be represented as a package. specifically,
package exclusions can not be adequately represented with exclude
dependencies. these need to be fed directly to the solver. if we
wanted to represent these in packages we'd have to invent a new
dependency actions or attribute. i initially prototyped this and talked
to bart about it and he just recommended that i feed exclude
dependencies directly to the solver, which is what i'm currently doing.
> lines 429-436:
> This could be greatly simplified by simply stating that the special
> system packages will not have a publisher. That is simpler and
> works better than having a special string constant value for the
> publisher. That also fits nicely with the transport framework
> since a publisher is required to perform transport operations, so
> simply checking for "if pfmri.publisher" is faster and we avoid
> the memory usage for the publisher string and parsing. Every FMRI
> normally has a publisher, so you can pretty much be guaranteed that
> in any case where you'd care, you can rely on simply checking to see
> if an FMRI has a publisher.
>
sure. i was never wedded to the name "none", and if this can be made to
work no problem. it's just that there's a lot of code that assumes a
publisher exists, so going this route might end up requiring more code
changes that defining a special publisher. (bart had suggested using an
invalid character in the publisher name to signify that it was
"special.")
> line 442:
> I wonder if one of the existing reserved namespaces that were
> proposed in May 2008 [1] could be used instead? In particular,
> feature, cluster, metacluster, or service? At the least, I'd
> like to see the name be a bit more specific than
> "linkedimage/constraints"; perhaps "parent-image-constraints"?
> I'm uncertain if the site/ namespace could be used here as that
> would seem to fit nicely with the purpose of this package.
>
i'd be more inclined to use feature. but once again, i don't really
feel strongly about it.
> lines 456-465:
> Why would the first package be empty instead of just making this an
> update from the out of sync version to the in-sync version of this
> package? Or alternatively, an uninstall of the old one and an
> install of the new one?
>
because as i alluded to before, you can't have an image with
inconsistent packaging installed. the sat solver can't transition you
to a valid state if your starting state is invalid. ie, if you have pkg
X installed and it depends on pkg Y.2, but pkg Y.1 is installed, you're
hosed and can't plan anything.
> line 481-491:
> This seems difficult and fragile for several reasons:
>
> - no guarantee that the operation in progress for the parent will
> complete successfully
>
so what? this data is used for planning operations initiated from
within parent images. (any operation initiated directly on a linked
image can't really take into account on-going changes in a parent image,
and eventually we'll need locking to enforce this.) for these types of
operations, we first do the planning across all the images and then
execute. so if the parent doesn't execute it's plan for some reason we
won't update the children.
> - does this distinguish between an operation being *planned* in
> the parent and one that is intended to be executed?
>
no. this package exists whenever an operation is being planned in a
parent image.
> - this implies that a parent image would always have to export
> (marshal?) its plan data during operations so that images that
> are linked to the parent (that the parent doesn't know about)
> can use this information
>
um. the parent can't really marshal data out to images it doesn't know
about. child images which the parent doesn't know about may be out of
sync wrt their content policy after any parent operation. subsequently,
this situation can be detected and corrected with a linked-audit and
lineked-sync on the child image.
> lines 526-543:
> Why are "in-core" packages defined using a package attribute
> instead of an incorporation? In particular, forcing this to
> be a property doesn't seem very flexible since the definition
> of what an in-core package is could be drastically different
> depending on the use case. Is the intent that this property
> is only used in special dynamically generated packages?
>
> I think I'd rather see a list of incorporations that were used
> to define the sync policy. That would also allow pkg freeze
> policies in the parent image to be applied to the child image
> more easily.
>
> It also seems like the property values could be a bit more
> user friendly. I'd suggest:
>
> sync-cip -> minimum
>
> sync-all -> exact
>
> superset -> possible
>
ok. so this is one area that i think is all going to change. i'll send
out a separate email describing how i think this should work.
>
> lines 565-570:
> independent-minimal:
>
> - This is a bit confusing to me as it seems to overlap with the
> li-content-policy property above. I see that you referenced
> that, but what I don't understand is why this wouldn't *always*
> be the case. Specifically, what's the real difference between
> this value and "independent"? I would think that the linked
> child image's content-policy would have to always be honoured.
>
so the values of this property needs to change, probably to:
li-update-policy = { full | minimal }
full would be the same as the old independent and minimal would be the
same as the old independent-minimal. (there was a reason for the
"independent" bit, but that reason has been rationalized away.)
here's an example of the difference between full and minimal.
lets say "pkg image-update" is run in the global zone, and we make a
plan to update the gz from snv_140 to snv_141.
now, if li-update-policy = full, then we'll go to that child image and
do an image-update there as well. so we'll likely update the image from
snv_140 to snv_141. but also, since it's a zone and it may have
different publishers and software from the gz, we'll also check for
updates from those publishers. so if oracle is installed in that zone
and it comes from a special oracle publisher, we'll update that if
there are updates available.
now, if li-update-policy = minimal, then we'll go to that child image
and only update the minimum amount of software that is required to keep
us in sync with the changes happening in the parent.
so in the end, the content policy is always honored, but this just
determines how aggressive we are with child updates when an image-update
is done an a parent image.
> lines 572-579:
> Will administrators be able to leave images "linked" but simply put
> them into a "disabled" state of some sort? I can definitely see
> administrators wanting to temporarily defer updating a linked image
> because of problems without preventing update of the parent image.
> I don't think simply unlinking it is the right solution as that
> means you lose the location information (which is valuable). This
> is not for zones obviously as you can simply detach those, which I
> assume simply puts them into an equivalent offline state or they're
> automatically skipped even though they remain linked.
>
this paragraph needs to change. currently you will be able to have
child images that are not in sync with a parent.
normally, if a child is in sync all operations that modify the contents
of the parent image get propagated to the child. but, if a child is out
of sync then we won't bother to propagate those operations.
also, there will be a couple ways to deal with operations on a parent
image that can't progress because of a child image.
first, i'm planning to provide linked-detach.
second, i'll provide a -i and -I flag to pkg. "pkg -I ..." will ignore
all linked images for a given operation. "pkg -i <linked-image-name>
..." will ignore a specific linked. if these flags are used then it's
likely that the child linked image will simply be out-of-sync after the
operation completes.
> lines 589-595:
> While this list fits the current definition and model of client
> operation execution, this will be changing in the near future. In
> particular, there are may be multiple data retrieval phases which
> means that you can't rely on this model to determine what can and
> cannot be done during different parts of an operation. More on
> that below in my comments for 610-636.
>
> As I mentioned above, I'd like to see a documented format for the
> plan serialization, and this may also be a good time to somewhat
> adjust how the client prepares and executes a plan to simplify the
> processes involved. Since what you've proposed here is basically
> marshalling an imageplan at different phases, I think this needs
> to be more generic so that we can use this functionality for low
> or restricted resource environments as well (which zones may be).
>
i've been using this for my prototyping, but i think it should go away.
i think it should be replaced with a control pipe that allows multiple
package processes to operate in sync. this would eliminate the need to
serialize out pkg planning data because it eliminates the need to have
pkg operations span multiple processes.
> lines 610-636:
> Instead of making these project private interfaces and to make them
> more generally useful, I'd like to see them become a bit more
> generic with the hope that they'd eventually be suitable for
> end-users. In particular, we already have a few RFEs open for
> being able to only perform the download portion of an operation,
> etc. With that in mind, I'd suggest the following changes:
>
> runid -> --plan
> Path to a pkg(5) plan file.
>
> plan -> --stage
> Specifying --stage without --runid
>
> Another question is where the plan gets stored when you run it
> with the --stage option? --runid seems too magical since it
> requires that the plan be stored within $IMGROOT, and I'd like
> for the plan to be retrievable from anywhere. If that's not
> desirable, can you expound on why?
>
> Finally, I'd also like to see the stages changed a bit to be more
> general in light of my earlier comments about multiple download
> phases and to be a bit more user friendly:
>
> default
> I'm assuming this implies that if --plan-id *is* specified,
> then the client will resume from the last point in the
> plan and this just continue until completion of the
> operation.
>
well, the "default" is the value used when --plan isn't specified. so
it means do all the stages.
> pkgs -> evaluate
> Just evaluates the operation (may trigger metadata
> retrieval in the v0 repository case). This is enough
> to determine what will be installed but not the size of
> the operation (disk space) or how many actions will be
> involved.
>
> actions -> prepare
> Prepares for package content retrieval and operation
> execution. This will retrieve package metadata (manifests).
>
> download
> Retrieve package content required to execute operation and
> exit.
>
> execute
> Execute the operation.
>
while this would be cool, i'm still planning on nuking all the current
"--plan" stuff. and while i do still need to convey planned constraints
to child images, once again, it's a subset of a full plan. so i don't
think it really lines up with the complete plan your talking about
above.
> lines 643-784:
> I think verb oriented subcommands are easier to remember and type;
> that also fits with our existing subcommand naming pattern.
>
sure, all of them except for image-update, not to be confused with
upgrade-image. :)
> Also, why require the -l option for specifying the identifier of
> linked images? Our other subcommands simply accept positional
> operands instead. Obviously you need to use an option for image-
> create, but the other subcommands don't really need one.
>
so in prototyping i've settled on the following convention:
pkg linked-audit
- assume the current image is a child and audit it.
pkg linked-audit -a
- assume the current image is a parent and audit all
it's children
pkg linked-audit -l foo
- assume the current image is a parent and audit it's
child named "foo"
this convention will apply to the following subcommands:
linked-export
linked-sync
linked-detach
this convention (sans the -a option) will apply to the following
subcommands:
linked-property
linked-set-property
> So with the above in mind:
>
> linked-list -> list-linked
>
> linked-sync -> sync-linked
>
> linked-unlink -> unlink-image
> No corresponding "link-image" subcommand? I'm aware that
> you account for it at image-create, wondered if post image
> create is also possible?
>
i'm going to rename this to linked-detach, and linked-attach (or
attach-linked) is also planned.
> linked-property -> linked-property
> linked-set-property -> set-linked-property
> Are these subcommands managing properties of the parent image
> that record information about a child (linked) image? Or are
> they reading and manipulating the general image properties of
> the child (linked) image?
>
if the -l <image> option is used then the operation applies to the
specified child image. if no -l option is specified it's assumed the
current linked image is a child and the operation applies to it.
> image-create
> What characters are allowed in the linked image name?
>
> It seems like being able to name linked images can lead to
> naming collisions. I'd like to suggest that this is strictly
> a human-readable "alias" for the image. And that you also change
> images to have a unique "id" (a UUID specifically), so that in
> the event that there is a naming collision, you can still
> perform operations on an image using it's unique ID.
>
> Alternatively, you could leave it as "name", but I'd still like
> to see images have a unique ID that we could rely on instead.
> Again, I wonder if the name here is a property of the linked
> image being created, or something that the master is recording.
>
> If the name is a property recorded in the linked image, then
> I'd also suggest that the parent may want to reference linked
> images only by their unique id & path to allow the name to be
> changed at any time.
>
my current thinking is that a fully qualified linked image name will be:
<linked-image-type>:<linked-image-name>
where the constraints on linked-image-name are type specific. if there
are no name collisions then a user can just specify <linked-image-name>.
if there are collisions a full name needs to be specified. so some
examples would be:
zones:<linked-image-name>
default:<linked-image-name>
user:<username>,<linked-image-name>
for zones, the linked-image-name would have to conform to the
restrictions on zone names.
for user images, the username would have to conform with what's
specified in passwd.4 as a valid username format.
for default images and for the user images, the linked-image-name would
probably have a conservative format similar to usernames and zones,
say:
- case sensitive
- first character must be alphabetic
- can contain alphanumerics plus hyphen (-), underscore (_), and
dot (.)
wrt who's recording the name, that will vary based on the type of linked
image. for images that the parent is not aware of, obviously that
information will be recorded in the child. for zones that information
will be stored in zone configuration files.
wrt uuids, i'm not to keen on using them because of the fact that the
place where we store information is distributed. for zones images, all
our information is stored in the zones configuration files. so there
really is no single authoritative place where we could store these uuids.
> lines 766-771:
> In the past, when I've suggested global options like this, it's been
> suggested that they be moved to the subcommands they actually apply
> to instead. For example, I doubt this option really applies to the
> refresh, info, or list subcommands of pkg(1). I'd also like to
> suggest that the option name be --ignore-linked or --skip-linked
> and can be specified multiple times.
>
sure. currently it was just easier to prototype it as a global option.
ed
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
