Stephen, Thankyou for this. Lots inline, although one general comment I have is that it's quite a muted case for such a significant change.
On Tue, Feb 26, 2008 at 7:23 PM, Stephen Hahn <[EMAIL PROTECTED]> wrote: > > To begin the design phase on > > http://opensolaris.org/os/community/on/os_dev_process/#development-process > > which involves the production of material for ARC review and test > plans, among other things, we need to exit the idea phase. (The > diagrams, while accurate, don't really allow any understanding of how > the phases actually overlap...) The "one pager" is the typical > document that announces to ARC the intent to pursue a project. > > I've written an initial draft and placed it for review here: > > http://cr.opensolaris.org/~sch/pkg-arc/ > > (It's also included below.) > > Comments welcomed. Once we iterate a bit (but not indefinitely, as > it's an introduction/announcement), we can send it into the ARC > machinery, which will assign various numbers, etc. > > - Stephen > > ---- > > Template Version: @(#)onepager.txt 1.31 07/08/08 SMI > > This information is Copyright 2007 Sun Microsystems > > 1. Introduction > 1.1. Project/Component Working Name: > > pkg(5): image packaging system > > 1.2. Name of Document Author/Supplier: > > Stephen Hahn, Sun Microsystems > > 1.3. Date of This Document: > > 02/26/2008 > > 1.4. Name of Major Document Customer(s)/Consumer(s): > 1.4.1. The Community you expect to review your project: > > Install and Packaging CG > > 1.4.2. The ARC(s) you expect to review your project: > > PSARC > > 1.5. Email Aliases: > 1.5.2. Responsible Engineer: > > [EMAIL PROTECTED] > > 1.5.4. Interest List: > > [email protected] > > 2. Project Summary > 2.1. Project Description: > > The image packaging system, pkg(5), is a portable software > packaging and delivery system intended to allow efficient, > observable, and controllable transitions between known > configurations of software content. At a minimum, it is > intended to combine in a more usable fashion the functionality > of the current packaging and patching utilities used with the > historical Solaris releases. The project includes a set of > recommended changes to the existing software groupings--the > package definitions--in an attempt to produce a more rational > and flexible organization of the current components. > > 2.2. Risks and Assumptions: > > It is assumed that the legacy packaging system functionality can > be preserved to support compatibility of existing packages. It > is further assumed that, if migration and compatibility > practices are made available, that the provision of a new > packaging mechanism will be followed by adoption. > > It is asserted that the refactoring and renaming of the existing > package graph is not achievable with reasonable cost and > duration with the existing packaging/patching/installation > software. It is assumed that compatibility with the existing > graph can be preserved, to support the earlier assumption on > preserving legacy package operations. That's an assertion that some of us regard as non-obvious. Indeed, it is entirely unclear to me how the package graph is dependent on the toolset used by user and administrators. > It is assumed that binary software delivery is the preferred > mechanism, over source build-based delivery, for a significant > majority of the deployment and development needs associated with > the operating system. > > 3. Business Summary > 3.1. Problem Area: > > Deficits in the current packaging, patching, and installation > tool set affect potentially all parties interacting with the > historical Solaris releases and their successors. Such deficits > include: > > - lack of support for dependency-based retrieval during package > installation, Errm. pkg-get has done this for years. > - coarse and incorrect dependencies, limiting use for > construction of appliances or other specific-purpose systems, A deficit in the package metadata, not in the toolset. > - lack of versioning and control over change, Again, I regard this as having more to do with package contents that the tools. > - forced interactivity, Forced? The current system *allows* interaction. > - integration with virtualized systems, particularly patching > performance, This is one area where I feel there was a huge mistake - that packaging was bent (broken) to fit around zones, rather than leaving packaging to just -well - package, and getting zones to understand packaging. This is one mistake we shouldn't aim to repeat. And patching performance is pretty abysmal in essentially all contexts, and most of it has zip to do with the underlying packaging technology. > - lack of safety, Please define the term! > - high developer costs around package and patch creation and > maintenance, > > - lack of support for unprivileged package and patch > installation, It sorta works. That's a minor implementation detail, really. > - lack of awareness of ZFS and smf(5), Why would a software administration need to be that aware of these technologies? > - late or no correctness checking, and > > - minimal ease of use. I've used a bunch of packaging systems. Generally, anything that just works without effort is good. From that perspective, existing Solaris software administration tools, while having flaws, generally work. Some others have superficial attractions, but prove to either have significant weaknesses or have managed to wreck important systems causing me significant amounts of work. > An additional cost, beyond those associated with a specific > operating system, is the absence of a portable and efficient > cross-platform software delivery system for cross-platform (or > minimally platform-dependent) software. > > 3.2. Market/Requester: > > Distribution providers and software content providers utilizing > the legacy packaging system have requested substantial changes > to achieve greater control over maintenance costs and to > increase development efficiencies. Now, what sort if changes? > Various customers of the historical Solaris release have asked > for substantial capabilities not present in the legacy packaging > system. I've asked - over and over - for changes. I'm fairly sure they're not what you're planning to deliver in this project. Again, what sort of changes have been requested? > Finally, multiplatform packaging capabilities are of interest to > a number of software content providers. > > 3.3. Business Justification: > > See 3.1 and 3.2. > > 3.4. Competitive Analysis: > > Every major operating system vendor--and most upcoming new > vendors--offers a form of networked software delivery and > updates. Well known companies with such technologies are > Microsoft, Red Hat, Apple, and Canonical; new companies include > rPath. Non-profit entities with equivalent technology include > the Debian Project. > > 3.5. Opportunity Window/Exposure: > > The project team asserts that, for Solaris to remain competitive > in terms of software delivery functionality, Solaris 10 should > be the last Minor release to not offer a packaging system that > meets or exceeds the minimal needs stated in 3.1 and 3.2. Yes, but 3.2 doesn't actually specify any needs. > 3.6. How will you know when you are done?: > > In terms of basic capabilities, we can examine each component. > Project completion on the retrieval side can be measured by > achieving the capability of managing mixed content from a > variety of publishers, with potentially distinct entitlement > regimes. On the publication side, completion of the initial > project is reached once the goals around dependency and > correctness checking (and failure handling) are met for both the > server and the publication client. Finally, ease of use (or > familiarity) must match or exceed that of other leading > packaging systems. > > In terms of the product as a whole, we must be able to upgrade, > with some statement about limitations on fidelity, a system > installed using the legacy packaging components such that it can > be further updated using the image packaging system. > > 4. Technical Description: > 4.1. Details: > > pkg(5) is a network-oriented binary packaging system. Although > it will have on-disk representations for versioned packages, the > primary expected use for installation of software will be > between an intelligent client and one or more relatively simple > servers. > > Additionally, the project defines a client-server publication > mechanism, in which the client offers up transactions on > packages, and the server evaluates those transactions for > completeness and/or safety prior to making them available for > retrieval by clients. > > The initial transport will be HTTP and HTTPS, protocols around > which most sites have developed mature access policies. Support > for most common HTTP/HTTPS load-balancing, redirection, and > proxying techniques will be implemented, making the system easy > to deploy in a variety of scenarios. Additional transports may > be investigated during the course of the project or as future > work. > > The project does not define a default mechanism for building > software as part of the packaging process. The project team > believes strongly that software builds are a separate function, > and probably also agrees that different kinds of software may > require different build techniques. > > More controversially, the project, in an attempt to increase > system safety and to reduce developer burden, removes the notion > of arbitrary context scripting from packaging. (This removal > means that the legacy packaging system must remain on the system > for long-term compatibility.) Empirical evidence from the > prototype phase has so far borne out this decision. Now, to be absolutely clear here: you're saying that you anticipate that future Solaris (OpenSolaris) releases must supply and support both the shiny new system and the crufty old system for an extended period - 10 years or so? And that users and administrators are going to have to know - and use - both (potentially incompatible) systems for that period? > 4.2. Bug/RFE Number(s): > > As an example of the kinds of defects and RFEs intended to be > resolved by this project, we present the following selection of > bug IDs from the past 15 years: > > 1105830 pkgadd and pkgrm should be able to handle dependency ordering > 1149607 Package dependencies hidden within a cluster. Not on b.o.o > 1165888 RFE: allow non-root users to install software using the package > mechanis > 1184238 patches should be fully managed by package utilites Not on b.o.o > 1208431 pkgrm with no arguments defaults to all > 1249015 pkgadd requires root access > 4202113 pkginfo command is ridiculously slow Well, it's not that slow, and inappropriate use for the problem. > 4240078 pkgadd should not allow an intel package to install on Sparc and > visa-ve Sometimes you want to do this. I'm not sure you can simply say this is always wrong. > 4385316 RFE Support pkgadd of clusters Yep, we really need this. > 4480153 Improvements desired for pkg management Seem to be in the invisible comments. > 4762470 pkgadd: soft dependencies Not on b.o.o > 4795539 pkgadd should check dependencies of all packages provided on the > command That's dependency ordering again. > 4847723 rem_drv in preremove scripts should have consistent usage model > 4939605 grep-friendly pkgchk -l variant desired Not on b.o.o > 5012345 request for tool to list package dependencies I've written such a tool; the spc exists; there's a relatively simple shell one-liner that I keep forgetting where I've put it. > 6208580 pkgadd/pkgrm should be smarter about dependancies b.o.o is highly forthcoming, but the statement is obviously true > 6246595 Sun's package management needs improvement Another one that's fairly blank, although it's yet another dependency handling one. > 6491381 Create audit log for packaging and patch commads > None of the above are particularly exciting or difficult. It's perhaps a source of embarrassment as to why some obvious pieces of functionality haven't been fixed in 15 years. Now, why can't these problems with the current system be fixed? It wouldn't take long, and could easily be backported to all currently supported releases allowing all users and customers to benefit. > In contrast, > > 1181241 wants to split large binary across multiple floppies with pkgmk > > will not be addressed by this proposal. > > 4.3. In Scope: > > Package-service delivery and containment relationships. > > Package installation behaviour in virtualized environments. Why is it packaging's problem, as opposed to the virtualization technology's problem? > 4.4. Out of Scope: > > Specific operational scenarios for repositories operated by Sun > Microsystems. > > Provision of a GUI/BUI for package management. BUI? Ah, Browser UI. Yuk! > Specific package contents and manifests. > > 4.5. Interfaces: > > pkg(5) will present a substantial set of new and modified interfaces > to the core system. In particular, documented definitions of > > - retrieval client CLI, > > - publication client CLI, > > - administrative and server CLIs, > > - client metadata representations, > > - server metadata representations, > > - retrieval and publication protocol operations, > > - a dynamic language API, > > - package metadata conventions, > > - available package constituents ("actions"), and > > - package naming and versioning conventions, > > will be presented as interfaces introduced by this project. One important aspect that is still missing is a definition of a package file format. For widespread adoption, this is *the* crucial factor. > It is possible that some of the nominally private interfaces > associated with legacy packaging will be affected; at a minimum, > files previously delivered via legacy packaging will no longer > be tracked by the legacy system. This outcome could result in a > correctly functioning system that presents very differently in > terms of file-package membership when interrogated using the > legacy packaging API. I'm worried again - this sounds to me that the system will likely appear to be broken. > Various components of the project will be introduced at each > stability and/or commitment level. The components are being > engineered such that the public interfaces can be evolved > compatibly, once the initial development is complete. (In fact, > the prototype is expected to support this evolution during the > development phase.) > > The components are currently implemented in Python > (PSARC/2005/532, PSARC/2005/555, PSARC/2006/666). > > 4.6. Doc Impact: > > The project expects to provide reference manual pages for each > of the groups of interface identified above. Furthermore, the > project expects to provide a Developer's Guide to replace the > current Application Packager's Guide. > > 4.7. Admin/Config Impact: > > Substantial new capabilities in software installation will > become available. > > A related project to produce a package manipulation GUI is being > pursued. > > 4.8. HA Impact: > > None known; dependent on specific impacts of legacy packaging on > these capabilities. > > 4.9. I18N/L10N Impact: > > Commands will require localization, as will any publically > committed library or equivalent APIs. > > 4.10. Packaging & Delivery: > > All package, cluster, and metacluster boundaries will be > examined in the course of the project. > > The primary upgrade mechanism for operating systems using pkg(5) > will be achieved via pkg(5) components; this mechanism is > expected to replace the current standalone upgrade and > LiveUpgrade paths. The replacement is expected, in concert with > the SnapUpgrade project, to present capabilities that equal or > exceed those of LiveUpgrade. -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/ _______________________________________________ pkg-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
