This seems like a good start at defining the source-side metadata that needs to be assembled to add good automation to the workflow of building the distro.
Without worrying yet about the representation of the data you propose for the .sauce files (we can paint the bike shed later), I'd like to think about workflow. The purpose of this metadata you have described is, after all, ultimately distribution developer workflow. When we created the Conary source management side, the state of source code control was remarkably different than it is today. Conary's source management was meant to have source-code-control ideas in it, and one of the points of contention within rPath was how much effort to invest in it as source code control and how much functionality to allow/encourage. Remember that we designed this when CVS was still the dominant source code control system, and the one with which we were most familiar. (We tried moving to Subversion and it ate the repository for lunch, and we used commitmail to revert to CVS.) The basic workflow was in place long before we switched to Mercurial for source code control. I don't think we should try to graft more complicated source code management ideas on top of Conary. I think we should move aggressively to use modern source code control for the source side of building a distribution, and consider conary :source components as records of what was built, for reproducible build purposes. I would like to do all our source management in Git. While I found the mercurial interface easier to teach, I have over the past two years become a strong advocate of Git, despite its UI oddities. I don't want this thread to become a discussion of Git vs. Mercurial; for now I'm just going to say "Git" and we can later discuss Git vs. Mercurial if anyone wants to. This does not mean that all recipes are factories. We can use CI to check source from Git directly into conary source components (push) instead of having conary fetch from Git (pull), which might make packages easier to understand. But what this does mean is that your proposed "spoon" command must remain a separate command that can run in the context of a Git repository that represents the contents of a source package. I'm not arguing against changes to Conary. For this to work, we need to be able to run cvc builds from a Git repository, for example. It might be that the spoon command (by whatever name) ends up included in the Conary package. And the interpretation of that metadata that is specific to the conary build process would be done with additional conary policy. Now, one of the things that would be easier if we used Git for controlling source packaging is that people who already know Git can easily clone a Git repository, make a change, and then send a pull request for their change. This means that the representation of the metadata should be chosen to make source code merges easy. This means that you want a representation that is unlikely to put unrelated changes in close textual proximity to each other. The easiest way to do that is to separate the changes by the part of the process they control, and put them in different files. So one file controls the build process (e.g. targets and flavors), one provides policy-related information (licensing, tags, and generally what you have described as "packages"), and one provides information that is primarily useful to the distro developer (homepage, aliases). (With regard to licensing, I think that Conary should use policy to discover licensing; it should differentiate licensing it infers from licensing it is told about, because the inferred licensing is a hint and can't deal well with things like exclusive choice of license. But that's also another topic.) Finally, regarding making it possible for everyone to help, this is another case where "fork me on github" is a powerful way to make collaboration work. I think we mostly agree; as important as I think these details of using external source management and breaking apart the files are, they are suggested means to accomplishing the goals you have laid out. On Thu, Feb 07, 2013 at 02:32:47PM +0000, António Meireles wrote: > As promised, yet again, yesterday, and way - way before to Mark, here we go. > > originally the conary tools ecosystem does not provide ways to allow, on > its native building toolset, the same kind of granularity, one got used to > on the deliverables side. > This is not a big issue on small projects but on large ones, turns things > less simple, and less instrumentable, than should and could be. > > the model detailed bellow is an attempt to close this limitation. > > each conary package :source file would become required to have a matching > .sauce file in a well determined xml format. [1] > > that .sauce file, albeit hand editable, would mainly be manipulated via a > command-line tool, spoon [2], (existing initially as a separate program, > see prototype written by Mark) with the intent of later get it > incorporated into cvc, from conary. > > the basic structure would be ... > > <?xml version='1.0' encoding='utf-8'?> > <recipe type="..."> > <source> > ... > </source> > <targets> > ... > </targets> > <flavors> > ... > </flavors> > <packages> > ... > </packages> > </recipe> > > > the <recipe> type is mandatory and will be one of the following: > > - default > - a regular recipe > - redirect > - a redirect recipe > - factory > - a factory recipe > - superclass > - a superclass recipe > > the <source> block is mandatory, whatever the <recipe> type defined. > > a typical <source> block would look somethink like ... > > <source> > <homepage>http://</homepage> > <licensing> > <license>MPLv1.1</license> > <license>GPLv2+</license> > <license>LGPLv2.1+</license> > </licensing> > <tags> > <tag>bootstrap</tag> > <tag>core</tag> > </tags> > <aliases> > <alias distro="fedora">fedora_foo</alias> > <alias distro="ubuntu">ubuntu_foo</alias> > </aliases> > </source> > > > All stuff here serves primarily for (build) instrumentation purposes [3]. > > <homepage> denotes the formal, upstream, origin of whatever we are building > <licensing> picks as individual <license> whatever licenses we are using. > here we will take advantage of Linux Foundations' http://spdx.org/ work > and machinery. [4] > <tags> is our way to tag :source packages for any given propriety / > characteristic we want. [5] > <aliases> is a simple way to bridge our naming to a given source package to > the ones given elsewhere. [see practical use case bellow]. > > Why is this useful ? > > think 'cvc spoon source status' [please excuse me, in advance, my lousy > ASCII art capabilities] > > homepage: ... > source version - foo > licenses ... > tags ... > flavors > arches > on file system > - foo+1 [uncommitted] > compiled versions > - in devel - foo [committed in ...] > - in qa - foo-1 [committed in ...] > - in production label - foo-5 [committed in ...] > version in fedora-18 ... > version in fedora rawhide ... > version in RHEL7 ... (as bar..srpm) > version in ubuntu 13.14 ... (ubar) > (...) > > > this gives one one instant overview about the status of any given source. > status in our side, and regarding the overall distro one. [things do not > exist in vaccuum.] > > [we can even put in conaryrc options to fine grain what we want to display > above, which labels to lookup, 3rd party distros to compare, etc ...] > > More on this in a bit. > > Now let's look at the <targets> block. > > a typical one would look like ... > > <targets> > <target>"x86"</target> > <target>"x86_64"</target> > </targets> > > > and basically it just states that the package is to be built against those > two targets. [policy would enforce this by blocking a package to be > built/committed, even if it builds, on something not white-listed] > > In a simmilar fashion we have the <flavors> block. [7] > a typical one could look like ... > > <flavors> > <flavor name="bootstrap"> > <flavoring>bootstrap,!cross</flavoring> > <flavoring>cross,bootstrap</flavoring> > </flavor> > <flavor name="!bootstrap"> > <flavoring>!cross,!bootstrap</flavoring> > </flavor> > </flavors> > > > Ok, this is not a too typical one, as most packages do not need bootstrap > flavoring. but is one that exemplifies the full power of this. > here, in this case, we state that this package has two flavors - > '!bootstrap' [6] and 'bootstrap'. Being that for the bootstrap flavor, in > same job, it will have to built twice with the flavoring speced in the two > <flavoring> entries. And, of course, for the !bootstrap case a single build > will suffice. > > Why do i want this, and the targets, thing ? automated build > instrumentation, continuous integration (think jenkins hooking), use this > to actually fine grain control all our the build machinery - end to end. > > And finally we have the <packages> block which is only expected to exist > when the <package> type is default. > > A typical one could look like ... > <packages> > <package name="foo"> > <licensing> > <license>MPLv1.1</license> > </licensing> > <aliases> > <alias distro="fedora">fedora_foo</alias> > <alias distro="ubuntu">ubuntu_foo</alias> > </aliases> > <tags> > <tag>core</tag> > </tags> > <descriptions> > <locale lang="en"> > <summary>Runtime libraries for foo</summary> > <description>This package contains runtime libraries for use by > foo:- the libfoo dynamic library, for use by applications that embed foo as > a scripting language, and by the main > "foo" executable - the foo standard library</description> > </locale> > </descriptions> > </package> > <package name="pkgspeced_from_foo"> > <licensing> > <license>LGPLv2+</license> > </licensing> > <descriptions> > <locale lang="en"> > <summary>extra foo for foo</summary> > <description>....</description> > </locale> > <locale lang="pt_PT"> > <summary>extra foo for foo</summary> > <description>....</description> > </locale> > </descriptions> > </package> > </packages> > > Here, with this block, our goal, is a bit more broader than just getting > more fine grained control on the :source side. goal here is to solve with > same shoot a bunch of different problems, but, on the deliverables part. > I want all built packages to have proper (userland) metadata about them, to > smooth user experience. short/long descriptions, origin, etc. think in more > refined conary search capabilities, think PackageKit integration, think on > lowering the entry curve - if we have packaged foo (which is named bar > elsewhere) a conary search 'bar' would find foo [ence the <alias> blocks > here]. Think in localization - we force all packages to have on each > <description> block a <locale lang="en"> one. but we 'd expect that over > time more languages get added. this is what the <description> and <alias> > blocks are mostly for. think, why not a fancy web browsing way to ship out > trees. > > the <licensing> one if omitted is assumed implicitly to be the same as > whatever is in the <source> block, and may also be used for policy > purposes. Same for the <flavors> and <targets> ones - We may want to > specify them in cases where, a given component/subPkg is only available in > a given flavor or target. > <tags> contents may also be used for policy purposes. say i have a 'text' > tag. policy will issue an error if any package tagged with 'text' ever > tried to depend on no text requires [anything that forced presence of X > libs]. <tags> will definitively be used for groups ... think in a > findByTag() action. shipping groups (even :cml) could be mostly rendered > through this almost automatically. > > Obviously populating initially the <package> blocks, specially getting all > the <descriptions> blocks right, even if only in english, takes a bit of > work, but, otoh, is a not too complex task where everybody is able to help, > and is essentially a one time hit. over time maintenance gets very cheap. > > And that's it. this is the 10000 feet overview and it should be good enough > to start a wider discussion. > > All the best, > > António > > > > [1] i 've picked xml here, because, well, because the format looked a bit > easier, visually on the eyes, than going to json, which would be another > valid choice. whatever we pick as the storage format, after processing i > suppose things will be pure json structures anyway, as that gives us a > cheap/simple and well established way to map things to python objects. > [2] spoon was picked to keep with the same culinary logic of 'cvc cook'. > it's not fully clear to me that this is the best final naming scheme :-) > [3] some stuff if not overrided in <packages> blocks reused there. > [4] getting SPDX integration right, on itself will take another email as > long as this one probably. > [5] To keep things manageable we will probably, in a way similar to what is > done with flavors, to predefine, from the start, the allowed tags > vocabulary. > [6] All packages, if undefined, have an implicit !bootstrap flavor. > [7] it's neat, and simple, to think of <flavors> and <targets> as fully > separate and independent things. I'm not sure, let me warn, that this way > renders all we need, long term. > > > > António Meireles > -- > Lead Developer, The Foresight Linux Project > http://www.foresightlinux.org > > > On Sat, Jan 26, 2013 at 4:21 PM, Mark Trompell <[email protected]>wrote: > > > > > Now to the new list: > > Hi > > I've been working on a little tool to help creating metadata files > > from commandline and make them accessible for conary through a python > > object > > I already got a bit foreward but real life put some timeconstraints on > > me, so I can't predict when I will be able to pick up where I stopped. > > Help is appreciated. I can give a bit of insight of what I did and > > Antonio should be able to explain what the tool should eventually be > > able to do. > > It's all written in python. And hosted at github[1]. > > Although the commandline tool is called spoon, feel free to stick a > > fork into it. > > Atm it uses lxml.etree which should be replace by pythons etree, so > > that the whole thing can (in the longer term) be added to conary > > eventually. > > > > Find me on this list or on irc if you need help. > > > > Mark > > > > [1]https://github.com/MarkTr/cnymeta > > > > -- > > Mark Trompell > > > > Foresight Linux Xfce Edition > > Cause your desktop should be freaking cool > > (and Xfce) > > > > _______________________________________________ > > Foresight-devel mailing list > > [email protected] > > https://lists.foresightlinux.org/mailman/listinfo/foresight-devel > > > > > _______________________________________________ > Foresight-devel mailing list > [email protected] > https://lists.foresightlinux.org/mailman/listinfo/foresight-devel _______________________________________________ Foresight-devel mailing list [email protected] https://lists.foresightlinux.org/mailman/listinfo/foresight-devel
