Hi Dave, Thanks for the feedback, comments/responses below...
On 05/21/10 07:54 PM, Dave Miner wrote: > On 05/19/10 10:34 AM, Darren Kenny wrote: >> Hi, >> >> We would like to ask people to please review the Data Object Cache design >> document, which can be found at: >> >> http://hub.opensolaris.org/bin/download/Project+caiman/DataObjectCache/DataObjectCache%2DDesign%2D0.5.pdf >> > > Overall, a very well-written document. But, of course, I have comments :-) Thanks, and I'm not surprised there are comments... > > 2.1, third bullet: s/system/application/ Sure, I just used the term system as referring to the complete mechanisms, but Application probably makes more sense. > > 2.2, last bullet: Would it more correct to say that it's the job of the > application to define the object hierarchy, and hence the data tree, and > that DOC is really just providing the infrastructure for the application > to do that? As written the DOC seems to be perhaps more omniscient than > you mean for it. Well, not really, mainly since it was thought that the DOC could make some use of the DTD / Schema (at a high level at least) to correctly structure the XML generated from the DOC. At least this is something we discussed with Sarah. The Application doesn't really know as much about the XML any more, but instead this is being deferred to each object in the DOC to know about their own area - e.g. Targets know how to map Targets from/to XML, etc. By the DOC making use of the Schema knowledge it is able to put the XML generated by various elements of the DOC into the correct places in the Manifest... At least that's the theory... This is largely being facilitated by the move towards DC and AI sharing the same Schema, but if there are differences we are considering passing flags to the to_xml(), and maybe from_xml() (but I think this should be less necessary) to allow for differences e.g.: DataObject.to_xml( manifest_type = MANIFEST_TYPE_AI ) > > 3.2 (sub-bullet of bullet 4) regarding the pickling requirements, is > there a specific reference or more concrete example that you could > provide to help ensure our object implementors get this right? The document at: http://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled explains what isn't out-of-the-box able to be pickled, but it essentially boils down to: The following types can be pickled: - None, True, and False - integers, long integers, floating point numbers, complex numbers - normal and Unicode strings - tuples, lists, sets, and dictionaries containing only picklable objects - functions defined at the top level of a module - built-in functions defined at the top level of a module - classes that are defined at the top level of a module - instances of such classes whose __dict__ or __setstate__() is picklable. So essentially if you stick to normal Python types you should be fairly safe. > > 3.2 bullet 5: Are there things we can do to help object implementers > meet the schema consistency and re-creation constraints? The hope is that the person implementing a given object, will be the most familiar with what the schema dictates for it's own XML equivalent, so it shouldn't be that much of an issue if they focus only on their specific area. Of course that doesn't mean there isn't something we could do to help, but I'm honestly not sure what we could provide other than a possible breakdown of the Schema for each area / checkpoint. > > 3.3 Is there a particular reason we need to constrain these to > Consolidation Private? (In reality, since install is not really > intended to be a separate consolidation, I'd prefer we avoided > consolidation and went with either Project Private or one of the public > levels). Are you intending a later update (once you've gone further > into implementation) with an imported interface table? There isn't really any good reason to constrain it to consolidation private, I wasn't really sure what was best for something like this. Project Private would seem TOO constrained, in that it's been mentioned that other it might be something people outside of the install team might want to utilize for adding another checkpoint (maybe?). Public would seem TOO open, possibly restricting things going forard. Maybe Uncommitted would be a better middle ground to require contracts for anyone wishing to use the interface outside of the project. I would be fearful that having it Committed would be promising too much at this point in time, but I would hope it would get to that point eventually after a couple of iterations. I totally open to peoples preferences here - but PSARC approved i/fs would seem to prefer the Committed option to Uncommitted. As for imported interfaces, it wasn't in the original document template we had, so really didn't cross our minds - but we should be able to add it, for what we know so far, but until implementation nothing really is set in stone. > > 3.4.1.1 Is there a reason not to require that names within a class be > unique? Not being able to depend on this seems to make some of the > other interfaces where retrieval/deletion can use names less useful. One reason that we didn't restrict this is in the case of Targets, where you may have something like: Targets TargetDisk [c0d0] Partition [p0] Partition [p1] Slice [s0] Slice [s1] ... Slice [s7] TargetDisk [c2d0] Partition [p0] Slice [s0] Slice [s1] ... Slice [s7] As you can see the names in this case (partitions/slices) wouldn't be unique in themselves, but would only be considered unique if you include the context of the parents, i.e. c2d0/p0/s1. I suppose we could ask that the name would be unique in the any given child list - but I don't think we could ask for it to be the case in the complete tree of objects. This could also open up the ability to refer to children using a dictionary, which might be useful... > > 3.4.1.2 Seems like it would be convenient to have a variant of > delete_children that can take a list (such as a list returned from > get_children). That is certainly possible, and we could also make this happen directly by allowing for the arguments to be a tuple, e.g.: delete_children( object1 ) delete_children( [object1, object2, ...] ) rather than seperate methods, and just handle this in the implementation itself. This could also be useful to apply to the "add_child" method, making it add_children()... > > 3.4.1.3 to_xml(). I like the potential choice to have a parent generate > a tree for its children, but I'm not sure how a general child class > would know to return None if it were implemented to normally provide its > own representation; especially if the parent would like to use the > child's to_xml() to assist in its aggregation. Should it perhaps be the > case that to_xml() also returns a boolean that indicates whether descent > within this object's subtree should continue? Should this also apply to > can_handle()/from_xml() so that the behavior can be fully symmetric? This is certainly possible to do. I'm honestly still delving into this area in more depth to see what the best solution would be. But my thinking on it is that if it's likely that the parent object would do the XML generation to include it's children, then most probably the case that the child wouldn't ever generate XML in itself. Of course, there's always some exception - I've just not thought of one yet... If we're to allow for such a case, it may be better to have an method like "generates_xml_for_children()" which returns boolean - I just don't like methods that return tuples of values as an interface. So this would make it more like: if not obj.generates_xml_for_children(): for child in obj.children(): ... The default implementation would return False - and this always traverse the children. > Finally, can you expand on the factors that are important to consider in > the decision between static and dynamic registration? My assumption > would be to lean strongly towards dynamic for flexibility of this > infrastructure, but I'm guessing there are factors that I'm not considering. I'm still thinking about this, but I think the main issue with static registration is that it means you need access to the source code to update the static registration file, which may not always be the possible. I would certainly prefer dynamic myself, the question then is just how dynamic we should be. One case I've been looking at is where on import of a module, it's __init__.py would call something like: DataObjectCache.register_class( MyClass ) for each class that can be used to import XML in a module. This works quite well (I've source code that tests it), but the main issue is that something needs to import the module... But I'm thinking that the Application, in most cases, will already be doing this, but maybe there are cases where it doesn't... In this case, we would need to consider something like a "signature" for a module which says that it's an install module with objects that can import XML. This would then require us to traverse the PYTHONPATH to find such a signature. This latter option introduces an time penalty at start up, but this may be offset by the flexibility it provides. would require A signature that I would be thinking of is a special file like: __install__init__.py which we could search for, and if it's found we would load that file and execute it - it would then contain the register_class() methods... I've still to look into this in more depth, and was intending on doing it as part of the implementation, but maybe I should pick one now... > > 3.4.2.1 A singleton here seems somewhat controversial to me. Why isn't > it the application's responsibility to control this? An alternate > formulation that I think accomplishes the goals here is to have the > application provide the Engine with an instance, and the Checkpoint > objects can always get it from their associated Engine. Are there cases > that this doesn't work for? (I didn't attempt to map this to all the > use cases so I'm not necessarily asserting it will, but it seems the > more natural solution to me so I'm wondering if you considered it). During the prototype phase we did try something like this, having the engine as the central point for getting a reference to the DOC, Logging, etc. but it presented more problems where everything had to access the engine to get a pointer to an instance, etc. - so instead we came up with each of these being singletons in their own right, so that they could be accessed from anywhere simply by calling XXX.get_instance() - but with the one caveat that something (in this case the Application) needs to create the first instance specifically - to ensure that ordering, etc is correct. > > 3.4.2.4 Wouldn't clear() be useful in applications like the interactive > installers where the user might go back to the first step in the parade > of screens (might be useful as a variant of Use Case 1)? Also, I didn't > grok what "dump( indented )" is supposed to mean? True, clear() could have many uses... As for dump() it's mainly for use in development or debug information to generate a "simple view" of the DOC at a given point in time. So you would get something like: DataObjectCache [root] Targets [DISCOVERED] Disk [c0t0d0] Partition [p0] .... and it uses the str() function to generate this, so an object may better represent it's output if it wishes. > > Diagram 2, page 20 I don't think it really matters from your design > point of view (probably helps the diagram to consider it this way :-), > but I expect that the current ICT glob will be re-implemented as > individual checkpoints. OK, we did this one up fairly early, so it's possibly not totally reflective of the final design. > > Diagram 3, page 21: s/Parition/Partition/ Fixed. > > page 25, first paragraph regarding writing out checkpoints to the > manifest: Seems like we need a mechanism for the application to inform > each checkpoint of whether it's to be written out or not. Not sure > where that falls architecture-wise. We seed as the checkpoints themselves knowing this (see above) - and simply need to return None if they are not going to generate anything. The whole idea that we're trying to achieve here is that all knowledge about a specific part of the XML schema is kept local - in that it's kept with the implementation of a DataObject sub-class. This should mean that if there is a need to change an object in some way the schema changes (if any are needed) would be made in the same place, and by the same developer hopefully... > > 4.4.1 NOTE: Does this also imply that checkpoints should be more or > less stateless? I suppose that it does, yes - a checkpoint's operation should be determined by two things: 1) The input requirements - which should be well documented - and be passed in the DOC - e.g. for TI, the desired Target layout is defined in the DOC with an object like : desired = doc.get_first_child_by_name( Targets.DESIRED_TARGETS ) and it can compare if it wishes with the discovered targets as found by: discovered = doc.get_first_child_by_name( Targets.DISCOVERED_TARGETS ) (Which should be documented as the output of the TD checkpoint). 2) The Checkpoint data's arguments (e.g. varied TransferCPIO) might have several instances with different cpio arguments. NOTE: This isn't the actual checkpoint, but the means to create it. I'm actually thinking this might be better to refer to as CheckpointData or CheckpointReference, etc. for more clarity... > > 4.5.1 Should you clarify that controlling the writing of execution > status to a persistent store is a function of the application and/or > engine, not the DOC? Yep, I think based on Alok's comments we need to make it clearer here who we're expecting to call what... So the Engine is the one that makes snapshots of the DOC, and the Application is the one that controls the load (roll-back) to a specific snapshot. > > 4.8 One thing that seems missing to me is how the application can prune > the targets that will be dumped into a manifest to just the ones that > were selected? All the other discovered target data seems to be noise > in terms of using the manifest in the future. True. In a past discussion with Sarah, we considered that maybe the DESIRED targets would be the only one to be dumped, and at that, it would by a sparse-tree, containing only the objects to be modified, which would be similar to what would be in an XML Manifest. This would make the output to the XML Manifest simple enough - this is why the DOC (as described earlier) would need to have some control of the high-level generation - in that it would then skip the DISCOVERED targets altogether, and only write out the DESIRED targets... Thanks, Darren. _______________________________________________ caiman-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/caiman-discuss
