On 05/27/10 07:01 PM, Edward Pilatowicz wrote:
On Thu, May 27, 2010 at 11:17:29AM -0700, Shawn Walker wrote:
On 05/26/10 09:25 PM, Edward Pilatowicz wrote:
On Mon, May 24, 2010 at 01:39:43PM -0700, Shawn Walker wrote:
lines 10-34:
So are linked image types just *additional* types of images (e.g.
full, partial, etc.) or is this a subtype of an existing image type?
hm. so i didn't even realize that pkg had support like things for
partial and user images... looking at the code i see that
IMG_TYPE_PARTIAL and IMG_TYPE_USER are not implemented. so lucky for me
i don't have a lot of re-designing todo to integrate with these types
of images. ;)
User images are not "fully implemented", but they are implemented in
the sense that image type is actually being used. Notably, the
glassfish team uses that functionality already. What's missing from
the user image type is the linked image capability.
i thought that the only difference between user images and all other
images was the place where the pkg metadata was stored.
".org.opensolaris,pkg" vs "/var/pkg". if that's the extent of the
differences then i don't really care that much.
It also affects how users and group information is retrieved.
But nothing beyond that (currently) as far as I know.
...
I'd like to work with you to figure out some way we can avoid that,
or alter the design where being able to link to a parent image isn't
possible without some sort of explicit enabling of linking
functionality.
sure. although i still don't understand why you would want this. for
push based children you need to have write access to the parent image.
that's essentially "explicitly enabling". for pull based children i
The question is, why does it *need* write-based access to the parent image?
don't really see why you'd want to do this. if i wanted to have a
partial user image that was synced up to whatever is installed on
jurassic, then since i have read access to jurassic i should be able to
do this. i don't see why i'd have to ask the jurassic admins for
permissions first...
Err...I'm confused now. Above you said write access to the parent image
is needed. In what cases would a child need write access to a parent to
perform a sync?
...
- also, with this approach there is no good way to deal with out of sync
images. ie, images that don't adhere to their sync policy. you can't
represent out-of-sync images in the installed catalog because you'd
have an invalid state, which leads to a very unhappy sat solver.
I don't follow; you could have that problem either way. Can you
give an example?
what i'm describing above is the problem that i also mentioned below.
if an image is out-of-sync with it's constraints, you can't represent
those constraints via an installed package. if you do, then the sat
solver can't solve anything.
...
so really the zone needs to know about a subset of the information in
catalog.attrs and catalog.base.C. now admittingly, i could represent
that data in the same format as the current catalog.attrs and
catalog.base.C files, but i haven't seen any advantage to doing that.
as i mentioned before, currently i just have an xml file that lists the
information, but i'm not wedded to that, it's more an implementation
detail.
The advantage of re-using the catalog format is avoiding introducing
an additional project private format and the fact that a lot of
optimisation work has been done to efficiently store catalog data in
that format.
sure. and if performance turns out to be an issue we can work on this
intermediate data exchange format as well. hell, it could be the same
format as the current catalog. (doesn't really matter to me.)
to put some real numbers to this, currently for my testing, i'm using
a snv_136 parent image with redistributable installed. for linked
images with a minimum sync policy the current package sync list (which
is a text base xml file) is 34K. if i decide to sync everything, then
that goes up to 138K.
The JSON equivalent should come in smaller than that, but I'd need to
see what data is recorded first.
lines 421-427:
Since a linked image can realistically only account for the last
sync'd state of a parent image, it seems like the parent image
could simply provide constraints as a dynamically generated
incorporation package (manifest) as part of the export/import
process. That manifest could then be stored locally and treated
exactly like a package normally would be without any special logic.
Doing so also makes it possible for the normal memory management
that the client api uses internally to not have to have special
logic to marshal this information to disk (possibly repeatedly).
possibly, but isn't manifest dependency information cached in the
catalog? if so this would require re-writing the catalog in zones which
requires reading the catalog in zones. as i've pointed out before, that
would be bad.
The catalog has to be read in zones; again, I'm not following the
security/bad logic. A catalog is used to track the state of the
image, if we don't trust catalogs, then the client is useless.
The only way to know the state of a child image is to read its catalog.
correct. and operations which are initiated on a parent image should
never do this. only pkg processes running at reduced privileges or in
special environments (zones or scratch zones) should access child
images.
Don't see a problem then; I wasn't suggesting that the parent read any
of the child's image data. I think we're in agreement here.
Although, I'd point out that technically, it's not safe for a child
image to read the parent's for the same reasons you point out above.
Quite an impasse...
in case it isn't obvious, i'm really trying hard keep data flowing in
one direction. from the parent image to the child. i'm also trying to
keep the data flow simple. having pkg traverse directories in a zone
image and read data from that zone isn't safe. i tried to call this out
early in the document with in the "zones requirements" section:
- since zones are untrusted execution environment, global zone pkg(5)
operations should not be required to read data from non-global zone.
ie, any data flow required to support linked images must be from a
global zone to a non-global zone, and never in the reverse direction.
I don't believe I've suggested anything that would require a ngz to
read data from a gz. I've only suggested *how* to export the data
and the process to use to import it.
perhaps i'm not understanding your suggestions. i have no problems
changing the data format used to export/push data to clients. i'm don't
want to use the existing on-disk parent catalogs because in many cases
that exposes to much information. i'm perfectly fine with using the
catalog format to expose a subset of what's installed. if you've
suggested something else then i'm sorry but i've failed to understand
it.
Ah, I see the confusion now. Sorry. The catalog format is fairly
fungible in the information that you can store. But the base
information doesn't really expose anything at all--it's just a list of
packages and an *optional* subset of each package's manifest data.
I wasn't strictly suggesting that you use the /var/pkg/state directory
as is.
also, there is one more thing to consider. not all the linked image
constraints on an image can be represented as a package. specifically,
package exclusions can not be adequately represented with exclude
dependencies. these need to be fed directly to the solver. if we
wanted to represent these in packages we'd have to invent a new
dependency actions or attribute. i initially prototyped this and talked
to bart about it and he just recommended that i feed exclude
dependencies directly to the solver, which is what i'm currently doing.
Can you give an example of how exclude dependencies can't currently
be expressed adequately using depend actions?
ok. but fyi, the cases where this can happen are complicated, and
unfortunately i erased the drawing on my whiteboard where i figured out
when it was impossible to do this with existing optional and exclude
dependencies. so if my explanation below seems confusing it's probably
because i got it wrong and i'll need a couple more tries explaining it
to get it right...
the requirement to keep packages in sync is actually tied to specific
versions of a package. so for example, say we are looking at packages
that utilize a "cip" packaging attribute that indicates they need to be
kept in sync between the gz and ngz. pkg A.1 and A.3 may not have this
flag set, while pkg A.2 may have this version set.
so, if the parent image doesn't have A.2 installed, then how can we
express (via the existing dependency actions) that the child is allowed
to have A.1 or A.3? optional dependencies only allow us to say verion X
or newer. exclude dependencies allow us to say versions older than
version Y. so if you look at different versions of a package as a
timeline, you can combine exclude and optional dependencies to allow you
to select one specific window of package versions from that timeline,
but you can't use them to allow non-sequential packages version from
within that timeline.
another complication is that optional and exclude dependencies do not
look at package timestamps. they only consider package version numbers.
in my initial prototype i worked around this issue by adding a
"exactmatch" attribute to exclude dependencies. if this attribute was
set i didn't apply the "older than" clause that is normally used by
exclude dependencies. this flag also made exclude dependencies take the
timestamp into consideration. bart convinced me that instead of adding
this new dependency attribute, it'd be better to simply feed the list of
excludes to the pkg solver and trim them from the solution space before
running the solver, which is what i'm currently doing. for all other
constraints i'm using incorporate dependencies, which are work down to
the specified precision, which normally includes a timestamp.
We've talked about expanding the expressiveness of dependency actions,
but based on what you stated above, I have to agree that the existing
mechanisms aren't likely to meet this project's needs.
It would be nice to figure out a way to do that though (not this project).
lines 429-436:
This could be greatly simplified by simply stating that the special
system packages will not have a publisher. That is simpler and
works better than having a special string constant value for the
publisher. That also fits nicely with the transport framework
since a publisher is required to perform transport operations, so
simply checking for "if pfmri.publisher" is faster and we avoid
the memory usage for the publisher string and parsing. Every FMRI
normally has a publisher, so you can pretty much be guaranteed that
in any case where you'd care, you can rely on simply checking to see
if an FMRI has a publisher.
sure. i was never wedded to the name "none", and if this can be made to
work no problem. it's just that there's a lot of code that assumes a
publisher exists, so going this route might end up requiring more code
changes that defining a special publisher. (bart had suggested using an
invalid character in the publisher name to signify that it was
"special.")
I'd rather fix the code that assumes a publisher exists than
propagate that bad assumption. There's lots of code that works
without a publisher too.
The logic for just not having one at all is simple--it completely
avoids any potential collisions with an actual publisher and it
ensures that the behaviour that we want for these is generic and
consistent and doesn't rely on magic values.
so. is it possible to have any other packages installed on a system
which have no publisher? for example, if i install a package and then
and then? I assume you mean and then "remove the publisher". Doesn't
matter; installed packages *always* have a publisher. Likewise packages
you can install *always* have a publisher. Remember that the publisher
is just an identity string of who published the package. So what you do
with unset-publisher has no effect.
...
My understanding of what you had written was that you had one
package that represented the last known good set of constraints on
the image, another package with the new set of constraints on the
image, and then one to account in progress updates.
So, my belief was that the first package represented what was
already "installed" in the image, that is, its initial state. I
don't see how that could possible be invalid to start with since it
represents the state of the child image after the last operation was
performed.
My belief then was that the other two packages represented the state
you were upgrading to, which the solver should be able to handle,
and if they're invalid, that's because the parent image is in an
invalid state, at which point, none of this matters.
no. let me try to re-state this. we have three packages:
- constrai...@0,0-0 - this package is always marked as installed. it
represents the current constraints on an image iff that image is in
sync with it's current constraints. if an image is NOT is sync with
it's current constraints, this package is empty.
- constrai...@0,0-1 - this package never marked as installed. if an
image is in sync with it's current constraints, this package is empty.
if an image is OUT of sync with it's current constraints, this package
represents the current constraints and installing this package will
bring the image into sync with it's current constraints.
- constrai...@0,0-2 - this package never marked as installed. the
contents of this package represents the planned constraints on an
image and installing this package will bring the image into sync with
it's planned constraints.
now. here's an example. say we have an out-of-sync image wrt it's
current constraints. the constrai...@0,0-0 package will be installed
but it will be empty. so to sync this image we install the
constrai...@0,0-1 package. this updates the image (if possible) to be
in sync with it's current constraints. of course, once this operation
is done if we list the packages contents in the image we'll see that
constrai...@0,0-0 is still installed. but once the image is in sync,
the contents of constrai...@0,0-0 will represent the constraints on the
image.
Your explanation makes sense, but I still don't understand why this
can't be represented as an upgrade from one state to another. In other
words, why have all of these different packages and all of this special
empty/not empty logic.
It would seem more logical to me to simply have the installed version of
the constraints packages represent whatever the last set of constraints
were that were installed. Then, any constraints you want to apply are a
newer version of that same package (timestamp only?).
There's just too much magic here, so I assume I'm still missing
something. I'd like to chat with you next week about this in person so
that I can get a better understanding.
...
>>>> - this implies that a parent image would always have to export
(marshal?) its plan data during operations so that images that
are linked to the parent (that the parent doesn't know about)
can use this information
um. the parent can't really marshal data out to images it doesn't know
about. child images which the parent doesn't know about may be out of
sync wrt their content policy after any parent operation. subsequently,
this situation can be detected and corrected with a linked-audit and
lineked-sync on the child image.
So then, you're saying that usage case is one where we generate the
parent image state on demand instead of the active export we do
automatically when the parent knows what linked images it has?
yes. parent's won't know about pull based children. so pull based
children will have to generate their own linkage data.
Okay, this is the part that wasn't clear and is quite helpful.
Its still confusing to me that the child can have a "full" sync
policy as well as the parent. I must be missing something still.
Why wouldn't you just set the full/minimal behaviour in the child
image instead, or why can't the child get the same behaviour without
setting something in the parent image? It feels like this is some
sort of override.
for any given type of image, the authoritative policy configuration can
only be stored in one place.
in the case of zones, the policy will be stored in a zones configuration
file in /etc/zones/, entirely outside the packaging system. during a
push, this policy will get written into the child so operations on the
child can take it into account, but the child can't change that policy.
for default push linked images, the policy will probably be stored in
the parent in the cfg_cache file, but similarly to zones images, will
be pushed into child images, where once again it is read only.
for pull based linked images, the parent won't know about them so the
policy configuration will live in the child and can be changed in the
child as desired.
Ah, ok. The part I was missing was that in the push (export) case, the
properties are both in the parent and child, but in the child they're
just a cache of the parent's information. And in the pull (import)
case, the properties are the authoritative one since the parent has no
record.
Also, why require the -l option for specifying the identifier of
linked images? Our other subcommands simply accept positional
operands instead. Obviously you need to use an option for image-
create, but the other subcommands don't really need one.
so in prototyping i've settled on the following convention:
pkg linked-audit
- assume the current image is a child and audit it.
pkg linked-audit -a
- assume the current image is a parent and audit all
it's children
pkg linked-audit -l foo
- assume the current image is a parent and audit it's
child named "foo"
this convention will apply to the following subcommands:
linked-export
linked-sync
linked-detach
this convention (sans the -a option) will apply to the following
subcommands:
linked-property
linked-set-property
I'm still not thrilled with the idea of starting all of the
subcommands with linked-; I don't think image-update and
image-create alone set the examples for how all our subcommands
should be named.
I guess the -l behaviour makes sense.
sorry i was unclear, i'm fine with renaming them to the verb prefixed
form. also, if there are too many commands that i'm introducing and
someone thinks it's polluting the command line namespace i'd even be
fine with introducing a new pkg-linked command. this stuff doesn't
concern me as much as getting the actual design and implementation
right. :)
I defer to Danek; I've made way too many interface decisions I regret to
feel confident about direction.
...
where the constraints on linked-image-name are type specific. if there
are no name collisions then a user can just specify<linked-image-name>.
if there are collisions a full name needs to be specified. so some
examples would be:
zones:<linked-image-name>
default:<linked-image-name>
user:<username>,<linked-image-name>
for zones, the linked-image-name would have to conform to the
restrictions on zone names.
for user images, the username would have to conform with what's
specified in passwd.4 as a valid username format.
I don't think username should be in the image name. User images
aren't literally tied to a specific "user". usernames can also be
considered security-sensitive information, so I'd rather not have
them there.
well, user linked images are out of scope for my current proposal, so i
don't feel compelled to nail down a naming strategy. i was just more
listing one above as an example.
Ah, okay. Probably best to just omit user images from the document
entirely other than to say this proposal doesn't address them ;)
The more I understand about this project, the more useful it seems to
user images, but at the same time, the more it becomes clear that user
images need their own separate design cycle.
...
My suggestions for using uuids was that they are far less likely to
collide with the sort of textual names suggested above. The textual
names are great for mere mortals that have to identify these images
on the command line, but uuids are almost guaranteed to provide
disambiguation while what has been suggested above is far less
likely.
Of course, one could argue that the probability of a naming
collision is low to begin with...
one could argue that, but i wouldn't. i don't want to design a system
that can't deal with naming collisions, which are bound to happen
because the linked image name space, as currently proposed, is partially
out of the control of pkg. for example, currently zone linked image
names match the zone name configured with zonecfg(1m). so unless i want
to start aliasing names, or start including uuids everywhere, then at
least my current idea gives me a way to avoid collisions.
Oh, I completely agree about ensuring the system can deal with
disambiguation. I wasn't implying that adopting UUIDs would somehow
resolve it of that responsibility, just that it'd be less likely.
UUIDs are probably overkill though so just ignore me ;) If we find
collision starts to become that much of a concern we can always add
UUIDs later. I've probably over-engineered this problem...
I'm happy overall with what you've suggested for disambiguation.
Cheers,
-Shawn
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
