As promised, yet again, yesterday, and way - way before to Mark, here we go.
originally the conary tools ecosystem does not provide ways to allow, on
its native building toolset, the same kind of granularity, one got used to
on the deliverables side.
This is not a big issue on small projects but on large ones, turns things
less simple, and less instrumentable, than should and could be.
the model detailed bellow is an attempt to close this limitation.
each conary package :source file would become required to have a matching
.sauce file in a well determined xml format. [1]
that .sauce file, albeit hand editable, would mainly be manipulated via a
command-line tool, spoon [2], (existing initially as a separate program,
see prototype written by Mark) with the intent of later get it
incorporated into cvc, from conary.
the basic structure would be ...
<?xml version='1.0' encoding='utf-8'?>
<recipe type="...">
<source>
...
</source>
<targets>
...
</targets>
<flavors>
...
</flavors>
<packages>
...
</packages>
</recipe>
the <recipe> type is mandatory and will be one of the following:
- default
- a regular recipe
- redirect
- a redirect recipe
- factory
- a factory recipe
- superclass
- a superclass recipe
the <source> block is mandatory, whatever the <recipe> type defined.
a typical <source> block would look somethink like ...
<source>
<homepage>http://</homepage>
<licensing>
<license>MPLv1.1</license>
<license>GPLv2+</license>
<license>LGPLv2.1+</license>
</licensing>
<tags>
<tag>bootstrap</tag>
<tag>core</tag>
</tags>
<aliases>
<alias distro="fedora">fedora_foo</alias>
<alias distro="ubuntu">ubuntu_foo</alias>
</aliases>
</source>
All stuff here serves primarily for (build) instrumentation purposes [3].
<homepage> denotes the formal, upstream, origin of whatever we are building
<licensing> picks as individual <license> whatever licenses we are using.
here we will take advantage of Linux Foundations' http://spdx.org/ work
and machinery. [4]
<tags> is our way to tag :source packages for any given propriety /
characteristic we want. [5]
<aliases> is a simple way to bridge our naming to a given source package to
the ones given elsewhere. [see practical use case bellow].
Why is this useful ?
think 'cvc spoon source status' [please excuse me, in advance, my lousy
ASCII art capabilities]
homepage: ...
source version - foo
licenses ...
tags ...
flavors
arches
on file system
- foo+1 [uncommitted]
compiled versions
- in devel - foo [committed in ...]
- in qa - foo-1 [committed in ...]
- in production label - foo-5 [committed in ...]
version in fedora-18 ...
version in fedora rawhide ...
version in RHEL7 ... (as bar..srpm)
version in ubuntu 13.14 ... (ubar)
(...)
this gives one one instant overview about the status of any given source.
status in our side, and regarding the overall distro one. [things do not
exist in vaccuum.]
[we can even put in conaryrc options to fine grain what we want to display
above, which labels to lookup, 3rd party distros to compare, etc ...]
More on this in a bit.
Now let's look at the <targets> block.
a typical one would look like ...
<targets>
<target>"x86"</target>
<target>"x86_64"</target>
</targets>
and basically it just states that the package is to be built against those
two targets. [policy would enforce this by blocking a package to be
built/committed, even if it builds, on something not white-listed]
In a simmilar fashion we have the <flavors> block. [7]
a typical one could look like ...
<flavors>
<flavor name="bootstrap">
<flavoring>bootstrap,!cross</flavoring>
<flavoring>cross,bootstrap</flavoring>
</flavor>
<flavor name="!bootstrap">
<flavoring>!cross,!bootstrap</flavoring>
</flavor>
</flavors>
Ok, this is not a too typical one, as most packages do not need bootstrap
flavoring. but is one that exemplifies the full power of this.
here, in this case, we state that this package has two flavors -
'!bootstrap' [6] and 'bootstrap'. Being that for the bootstrap flavor, in
same job, it will have to built twice with the flavoring speced in the two
<flavoring> entries. And, of course, for the !bootstrap case a single build
will suffice.
Why do i want this, and the targets, thing ? automated build
instrumentation, continuous integration (think jenkins hooking), use this
to actually fine grain control all our the build machinery - end to end.
And finally we have the <packages> block which is only expected to exist
when the <package> type is default.
A typical one could look like ...
<packages>
<package name="foo">
<licensing>
<license>MPLv1.1</license>
</licensing>
<aliases>
<alias distro="fedora">fedora_foo</alias>
<alias distro="ubuntu">ubuntu_foo</alias>
</aliases>
<tags>
<tag>core</tag>
</tags>
<descriptions>
<locale lang="en">
<summary>Runtime libraries for foo</summary>
<description>This package contains runtime libraries for use by
foo:- the libfoo dynamic library, for use by applications that embed foo as
a scripting language, and by the main
"foo" executable - the foo standard library</description>
</locale>
</descriptions>
</package>
<package name="pkgspeced_from_foo">
<licensing>
<license>LGPLv2+</license>
</licensing>
<descriptions>
<locale lang="en">
<summary>extra foo for foo</summary>
<description>....</description>
</locale>
<locale lang="pt_PT">
<summary>extra foo for foo</summary>
<description>....</description>
</locale>
</descriptions>
</package>
</packages>
Here, with this block, our goal, is a bit more broader than just getting
more fine grained control on the :source side. goal here is to solve with
same shoot a bunch of different problems, but, on the deliverables part.
I want all built packages to have proper (userland) metadata about them, to
smooth user experience. short/long descriptions, origin, etc. think in more
refined conary search capabilities, think PackageKit integration, think on
lowering the entry curve - if we have packaged foo (which is named bar
elsewhere) a conary search 'bar' would find foo [ence the <alias> blocks
here]. Think in localization - we force all packages to have on each
<description> block a <locale lang="en"> one. but we 'd expect that over
time more languages get added. this is what the <description> and <alias>
blocks are mostly for. think, why not a fancy web browsing way to ship out
trees.
the <licensing> one if omitted is assumed implicitly to be the same as
whatever is in the <source> block, and may also be used for policy
purposes. Same for the <flavors> and <targets> ones - We may want to
specify them in cases where, a given component/subPkg is only available in
a given flavor or target.
<tags> contents may also be used for policy purposes. say i have a 'text'
tag. policy will issue an error if any package tagged with 'text' ever
tried to depend on no text requires [anything that forced presence of X
libs]. <tags> will definitively be used for groups ... think in a
findByTag() action. shipping groups (even :cml) could be mostly rendered
through this almost automatically.
Obviously populating initially the <package> blocks, specially getting all
the <descriptions> blocks right, even if only in english, takes a bit of
work, but, otoh, is a not too complex task where everybody is able to help,
and is essentially a one time hit. over time maintenance gets very cheap.
And that's it. this is the 10000 feet overview and it should be good enough
to start a wider discussion.
All the best,
António
[1] i 've picked xml here, because, well, because the format looked a bit
easier, visually on the eyes, than going to json, which would be another
valid choice. whatever we pick as the storage format, after processing i
suppose things will be pure json structures anyway, as that gives us a
cheap/simple and well established way to map things to python objects.
[2] spoon was picked to keep with the same culinary logic of 'cvc cook'.
it's not fully clear to me that this is the best final naming scheme :-)
[3] some stuff if not overrided in <packages> blocks reused there.
[4] getting SPDX integration right, on itself will take another email as
long as this one probably.
[5] To keep things manageable we will probably, in a way similar to what is
done with flavors, to predefine, from the start, the allowed tags
vocabulary.
[6] All packages, if undefined, have an implicit !bootstrap flavor.
[7] it's neat, and simple, to think of <flavors> and <targets> as fully
separate and independent things. I'm not sure, let me warn, that this way
renders all we need, long term.
António Meireles
--
Lead Developer, The Foresight Linux Project
http://www.foresightlinux.org
On Sat, Jan 26, 2013 at 4:21 PM, Mark Trompell <[email protected]>wrote:
>
> Now to the new list:
> Hi
> I've been working on a little tool to help creating metadata files
> from commandline and make them accessible for conary through a python
> object
> I already got a bit foreward but real life put some timeconstraints on
> me, so I can't predict when I will be able to pick up where I stopped.
> Help is appreciated. I can give a bit of insight of what I did and
> Antonio should be able to explain what the tool should eventually be
> able to do.
> It's all written in python. And hosted at github[1].
> Although the commandline tool is called spoon, feel free to stick a
> fork into it.
> Atm it uses lxml.etree which should be replace by pythons etree, so
> that the whole thing can (in the longer term) be added to conary
> eventually.
>
> Find me on this list or on irc if you need help.
>
> Mark
>
> [1]https://github.com/MarkTr/cnymeta
>
> --
> Mark Trompell
>
> Foresight Linux Xfce Edition
> Cause your desktop should be freaking cool
> (and Xfce)
>
> _______________________________________________
> Foresight-devel mailing list
> [email protected]
> https://lists.foresightlinux.org/mailman/listinfo/foresight-devel
>
>
_______________________________________________
Foresight-devel mailing list
[email protected]
https://lists.foresightlinux.org/mailman/listinfo/foresight-devel
