On Jul 14, 2013, at 11:23 PM, Baptiste Daroussin wrote:

On Sun, Jul 14, 2013 at 10:12:19AM +0000, Teske, Devin wrote:

On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote:

On 14/07/2013 06:48, Teske, Devin wrote:
Question: Where can I learn more about the actual format of what's in
the new tarballs? This is going to be important not for bsdconfig,
but $work (we have our own build platform; I'm going to have to
rewrite it from mastering PLIST files to mastering YAML MANIFEST
files and I want to know all the gritty details).

We do need a pkg-manifest(5) man page, which can double as a
pkg-tarball(5) page since the manifest file is where most of the
interesting bits are.

A pkg tarball is a compressed tar archive like so:

lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz
-rw-r--r--  0 root   wheel     530 Jan  1  1970 +COMPACT_MANIFEST
-rw-r--r--  0 root   wheel    6385 Jan  1  1970 +MANIFEST
-rw-r--r--  0 root   wheel   17567 Jan  1  1970 +MTREE_DIRS
-r--r--r--  0 root   wheel   19453 Jul  7 12:26
-r-xr-xr-x  0 root   wheel     629 Jul  7 12:26
-r-xr-xr-x  0 root   wheel     823 Jul  7 12:26
-r-xr-xr-x  0 root   wheel     678 Jul  7 12:26
-r-xr-xr-x  0 root   wheel    2558 Jul  7 12:26
-r-xr-xr-x  0 root   wheel     611 Jul  7 12:26
-r--r--r--  0 root   wheel     839 Jul  7 12:26
-r--r--r--  0 root   wheel   43432 Jul  7 12:26 /usr/local/include/pkg.h
-r--r--r--  0 root   wheel  727558 Jul  7 12:26 /usr/local/lib/libpkg.a
lrwxr-xr-x  0 root   wheel       0 Jul  7 12:26 /usr/local/lib/libpkg.so
-> libpkg.so.1
-r--r--r--  0 root   wheel 1227064 Jul  7 12:26 /usr/local/lib/libpkg.so.1
-rw-r--r--  0 root   wheel     187 Jul  7 12:26
[... etc ...]

Interesting. I notice that (while looking ahead to see a prefix: of /usr/local 
in the +MANIFEST), the tarball itself has files that include /usr/local in 
their path.

Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" _but_ 
"tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_install style) 
or is this the new way going forward?

There must at least be a +MANIFEST -- other meta data files are
optional.  +COMPACT_MANIFEST is a subset of the full +MANIFEST.  They're
both YAML documents.  +COMPACT_MANIFEST looks like this, for example:

name: pkg
version: 1.1.4
origin: ports-mgmt/pkg
comment: New generation package manager
arch: freebsd:9:x86:64
maintainer: port...@freebsd.org<mailto:port...@freebsd.org>
prefix: /usr/local
licenselogic: single
flatsize: 6311507
desc: |
New Generation package management tool for FreeBSD

- ports-mgmt
- libpkg.so.1
- libpkg.so.1
message: |
If you are upgrading from the old package format, first run:

  # pkg2ng

Nice. Very nice.

+MTREE_DIRS is a compatibility thing with the old pkg_tools.  It's not
needed in general as +MANIFEST can provide all that meta data itself.
It isn't going to be deprecated for at least as long as the ports tree
continues to support pkg_tools though.

Beyond that, the rest of the pkg tarball just contains a tar archive of
all the files, directories, sym-links etc to be installed by the
package.  Note that pkg(8) has no problem with creating an empty
directory for a package, unlike pkg_tools.


Now, while you can grovel through the details of pkg tarballs and
internal data formats like this, be aware: the format of the manifest
and the details of the meta-data included in the pkg-tarballs is subject
to change without warning as we develop pkg(8) further.  We only promise
API stability through the pkg(8) commands or for the libpkg.so library
functions; reports of breakage from usage outside those APIs will
receive little sympathy.


So just to give you a better idea of how we manage packages here.

We've realized that if you want to package for FreeBSD (in 9 and older), you 
could get by alright if you knew how to create a +CONTENTS file from scratch. 
For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awful lot 
like a SPECFILE (they are both in-fact YAML).

The only thing you need if you really want to package is a small subset of the
manifest written in YAML:

Have a look at what the ports tree do to create the minimalistic manifest and
what form it has and you will see how easy it is.

It is so awful and complicated that there are already a load of custom scripts
to create packages with pkg without using the ports tree. As an example:
 , but there is way more.

I for example wrote in the past a plugin for pkg to be able to parse and USE
RPM's specfiles as an input for pkg.

Cool. Wonder if anyone will end up using that. It does sound rather neat for 
(perhaps) people more RedHat centric that may want to re-use the same SPECFILE 
of their RPM to produce a pkgng file.

So rather than teach the people here how to use tools, I taught them what I 
think is more important... how to manage +CONTENTS, SPECFILE, and now 
up-and-coming, +MANIFEST.

The problem is that you compare 2 things that cannot be compared,

In normal light, yes... but when actually, we've essentially wiped the 
existence and knowledge of "source RPMs" from our minds as our build system 
imbues us with that luxury (the luxury of never needing a source RPM, never 
producing one, and never even thinking about one -- we go from SPECFILE 
directly to binary RPM and when we need to modify a binary RPM, we take a 
template SPECFILE and populate it with the metadata extracted from the rpm file 
itself (using "rpm" utility to extract said metadata).

the equivalent of pkg + the ports tree.

In normal case, yes... in our case no. Every single SPECFILE in our build 
system uses the exact same assembly procedure to produce [directly] a binary 

In fact specfiles are the equivalent of a given "port" !!! Are you writing rpm
files directly by hand? there is no SPECFILE inside RPM but rather a header
container metadata in binary format and a body which is a cpio.(gz,bz2,xz).

Correct, a binary RPM does not contain a SPECFILE. But, every piece of data 
that goes into the SPECFILE is contained in metadata extractable with "rpm -q 
-p RPMFILE --qf FORMAT" (iirc).

This is how we turn a binary RPM into a build-directory with a SPECFILE that -- 
when we say "make" -- produces the same binary RPM file with-or-without any/all 
changes to the flat [unpacked] hierarchy:


You can take a look at an example SPECFILE that's been generated from a binary 
RPM using the above script:


NOTE: In the case of the above URL, it turned out that the "sup" package wasn't 
available for RedHat EL 6 -- so we slurped in some other simple package and 
replaced its guts with that of a sup package compiled specifically for RedHat 
EL 6 (with fixes).

But the SPECFILE starts its life as a template for each/every unpack of a 
binary RPM (see below):


You are not creating the RPM file directly but rather create a SPECFILE, the
equivalent on FreeBSD is to create a port!!!

Incorrect; as you can see from the above, we *do* treat SPECFILE as a PLIST 
(aka +CONTENTS). We do build the RPM from the SPECFILE.

You're absolutely right that the SPECFILE *normally* contains logic on how to 
turn a vendor software X (in source form) into [first] a "source" RPM and then 
a "binary" RPM. But we've taken all that logic out of the SPECFILE and turned 
it into a dumb thing that just goes direct to binary RPM.

the +MANIFEST as used to be the
+CONTENTS are the equivalent of the RPM metadata.

Right -- we use "rpm"'s --queryformat parameter to extract the metadata.

We wrote them in YAML so that
they are easy to debug and that we can use third party well used and tested
parsers instead of writting our own.

I'm very thankful that the compiled pkgng package isn't as opaque as RPM. I 
think it's a good thing for transparency. I also like that you're porting your 
tools. pkgng has indeed a bright future and you've been making great choices.

(and there is no "but" -- just wanted to say "thank you")

I hope you don't see my dissection attempt as a way to subvert what you're 
doing. Quite the opposite, I'm dissecting so that I can learn in an effort to 

However, I'd be lying if I said I taught them *all* the gory details. In 
reality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS without 
the "@comment MD5:" entries. For SPECFILE, they manage the full thing except 
there's a small section that is the standard RPM bootstrap that is labeled as 
"do not touch".

>From what you posted of +COMPACT_MANIFEST was nice, except it lacks the list 
>of files.

The basic principle here is that if you have to maintain a list of files, and 
that list ends up being compiled into a +MANIFEST, why not just teach your 
peers how to read/write/maintain +MANIFEST files (in version control of course) 
so that there are never any mysteries as to how the package performs.

Have a closer look at pkg create, you will see that we are still able to use the
plist or +CONTENTS as a possible input for pkg create, it is even the main way
to create packages.

Ooo, nice. That will smooth migration (however, we it will be a while before we 
get onto 10.x+ so we've got some time -- meaning, I might favor a route that is 
legacy free.. agree?)

Once again, the main and most important requirement to build packages was to be
compatible with the ports tree, and the ports tree is using plist/+CONTENTS.

and pkg2ng does only works by using the +CONTENTS.

I know this sounds screwy... but (as with our other YAML files), it's really 
nice because (as is the case with SPECFILE), we version-control things as-close 
to the end-product as possible (take for example the case of "pre-install" or 
"post-install" et cetera scripts -- I'm sure a tool like "pkg create" or 
"pkg_create" or other could do a good job of assembling the pieces into the 
+MANIFEST, but then it's not being version-controlled as closely.

You can keep what you have already done.

Once again pkg does not have the equivalent of the SPECFILE because we have the
ports tree!

In Linux, rpmbuild will take a SPECFILE in the manner you mention (acting as 
the pre-cursor to the SRPM), but it will also take a SPECFILE to build a binary 
RPM (and in that last manner, the SPECFILE is not like the ports-tree as it 
expects binaries, not source, to build the cpio-ball).

In the manner we're using SPECFILE (the latter manner), it *is* like a PLIST or 

so we do not need to provide a custom package building system.

We could, as it is really easy, but we do not need so it is not there yet, as
said earlier I already wrote a pkg plugin that is able to build packages from a
RPM specfile, I also modified archlinux's makepkg to be able to output pkgng 
I did the same on slackare, etc. Here the limitation is not technical at all.

That's really nice and encouraging. As I do-say... it's remarkable and the 
future of pkgng looks bright ;D

The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
freebsd-current@freebsd.org mailing list
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Reply via email to