On Jul 14, 2013, at 3:43 PM, Craig Rodrigues wrote:

On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin 
<devin.te...@fisglobal.com<mailto:devin.te...@fisglobal.com>> wrote:

I assume that poudiere builds packages from ports.


That's not how we build package repositories here (and would expect that there 
are many more like us).

How do you build packages if you are not using FreeBSD ports?

Two spectrums...

When developer A says he needs xerces, I go to:

(for an active release, such as 8.4 or 9.1)

(for an in-active release, such as 8.1 or 9.0)

Meanwhile, when either developer B or integrator A says "I have software X that 
I want to package", we use "pkgbase" to build the FreeBSD package, the RedHat 
RPM, etc.

  Do you have your own Makefiles which resemble FreeBSD ports,
or are you doing something completely different?

That's essentially what pkgbase is. Except instead of building makefiles that 
build the 3rd-party software, we rely on the 3rd-party compilation process to 
produce the binaries and have a configuration file (pkgbase.conf) which 
describes the vendor layout.

The vendor layout can be anything and do anything and prepare anything. But the 
pkgbase.conf says "fileA" in the vendor layout goes to "src/fileA" (where 
"src/" looks exactly like the un-packed tar-ball, *minus* +CONTENTS -- which is 
generated from PLIST when you say "make").

Since we're using CVS, people that are familiar with CVS would think it's quite 
cute that it does a "cvs admin -ko" on files that are pulled from the vendor 
branch (resulting in keyword expansion to retain its vendor location -- so if a 
problem happens in the field, we know that the file that should be edited -- 
based on its keyword expansion -- is over in the vendor "depend" tree which, 
upon modification, will cause a cascading rebuild of packages for releases we 
actively maintain [through a sliding window]).

So it's very similar to ports, but we need the mechanism of being able to (for 
the benefit of developer A/B and integrator A above) freeze the software 
distribution of software X.

The ports tree manages that through a collection of centrally located distfiles 
(which acts as a fall-back for when vendor distfiles dry up). But that method 
doesn't work so well for us because when we find a bug we often fix it 
ourselves in our own local branch and then communicate the fix back the vendor 
(leaving it up to them to incorporate it). The next time around on our 
migration to a new release, *if* we have made a local patch, producing a custom 
version, *and* the vendor has not taken a patch, we'll _first_ re-evaluate the 
new version with a documented test-case or (if that fails) we'll keep the patch 

The ports tree moves too fast for us. And many times we find a bug, we end up 
performing cost/benefit analysis only to find that we can live with the bug 
*if* the problem has been fixed already. In rare cases, we'll merge a single 
patch from a future revision to get things going. But we *won't* just blindly 
take a new version simply because it fixes a bug (as it may introduce new bugs 
-- unwinding many man-hours of many testers and developers attempting to lead 
up to a release of a wholly contained full-working software entity).

I hope this helps to explain a little more about how we take FreeBSD releases, 
vet them, and them blast them out to thousands of machines over many customers 
after many months of testing.

Now to say that we don't use ports would be a lie. Sometimes a developer asks 
for a package that is not available on the release we're shipping. This 
happened recently. So I went into the ports tree, built the package on 8.1 
(which had 99% binary packages from 8.1-RELEASE -- see above URLs) and then I 
said "make package" and threw it in the repository.

ASIDE: For diligence, "make describe" iirc generates my INDEX entry to append.

We expect to start with an empty directory, go grab packages that we want (one 
by one) from FTP. Then put those tarballs into a directory. There's no 
additional step because we download the INDEX file to the repository too. Local 
repository is built.

What FTP site are you grabbing packages from, the FreeBSD ftp site (or mirrors) 
or one of your own?

We grab from either ftp.f.o or ftp-archive.f.o and plop into an NFS-based repo 
(which has a structure mirroring that of the ftp{,-archive}.f.o structure).

How are you generating the INDEX file?

Well, since we prefer to ship 100% with the packages that were cut with 
X.Y-RELEASE, the INDEX file that's up on the ftp.f.o site does fine (except 
when we have to inject things from ports -- for example, when developer A needs 
software X which isn't available on X.Y for the binary release packages -- 
in-which case we have "make describe"). However, as previously mentioned, the 
tool that mass-installs the architecture (and release) specific packages out of 
the repo was written by a different developer (before I came to this company) 
and *it* doesn't care about INDEX.

  Do you append to the INDEX file for every package which you sucessfully 
download from FTP,
or are you downloading a previously generated INDEX file?

The latter (except for rare cases that I want a newly-injected port package to 
be browsable/usable by sysinstall -- which uses INDEX).

  What does your local repository of packages consist of if you are
not using FreeBSD ports?

The goal each release that we cycle through is indeed to wipe the local 
repository completely clean of any/all packages that we ourselves do not 

But it just doesn't happen.

As previously mentioned, when we start to track a new FreeBSD release for 
development/testing purposes [in our labs], we start with binary release 

When we find that a package (either a trusty one that we've known and loved for 
over a dozen years; OR a new one that we're still feeling out) doesn't operate 
quite right, here's what we do...

We slurp it into pkgbase (it has a method for doing this; requires only a 
"./unpack.sh -o bad_package.tbz" -- which not only unpacks the tarball, but 
puts it in the specific origin sub-directory, reverses the +CONTENTS file into 
a PLIST, and assembles the Makefile system into the unpacked directory so that 
you could, if so-desired, go in and say "make" and get the exact same product 
as what you unpacked; after you say "unpack" you then go in and say "make 
import" and it imports it into the revision control system -- CVS for us).

Once slurped-in, we then tack-on a "_1" to the version in the PLIST file, go 
fix the problem, commit both the fix and the bumped version back to CVS. Then 
we say "make tag" and it tags the files with the version specified in the 
PLIST, and then finally... "make" which produces the *fixed* "bad_package.tbz" 
which can then be put into the repository. We then go tack-on "_1" to the 
version that is specified in the arch-specific config for our tool, and that 
architecture now installs (and will update-to) the fixed package.

Packages break. We choose to deal with it silently within our own release 
cycle. If we notice that something silly hasn't been fixed in the time it took 
us to jump to the next release, we often release our fix for it (because 
obviously nobody else came up with a better fix). But, more-oft than not, what 
happens is that the breakage was a silly thing and somebody fixed it right away 
(but we couldn't take that fix because it was post-release and we couldn't take 
the risk of bringing in other things). ASIDE: Even if we did take the fix from 
above, we'd have to have our own local system *like* what I'm describing so 
that we can maintain accurate audit-history for our clients that demand a 
break-fix mentality rather than a break-upgrade mentality.

But I want bsdconfig to work with local repositories without having poudriere.

You don't have to use poudriere to build a package repository.  It is just an 
optional tool that
is there for you to try out, and it works quite well when building a package
repository based off of FreeBSD ports.  If you don't want to use it, you don't 
have to.

I was *quite* sad when oldports.org<http://oldports.org> went away (not that I 
used it much, as I could always check out historical revisions of the port 
Makefiles, etc.). It offered a very simple service... download a tarball of a 
snapshot of any port for any given revision.

For poudriere (or ports for that matter) to work for something like $work, it 
would have to accommodate inordinate durations of being frozen *and* work well 
with a static versioning system.

And as I was typing that, I think I realized a way to do it...

Each release of FreeBSD comes with a "ports.tgz" distribution. Would I be wrong 
in assuming that this essentially represents the very snapshot-in-time of the 
ports tree in-which the binary release packages were built?

I think I've heard others mention that this is not always guaranteed to be.

That aside, I'm not so sure that it would be that beneficial to build the 
packages ourselves when we so-rarely (and prefer not-to) have the need to 
remaster packages. In other words, we like leveraging the time someone else 
took to compile the packages.

I think so too. But right now a *lot* of unanswered questions.

Possibly, but I think that you are supporting a workflow that people on this 
mailing list
don't fully understand.   If you could send out a separate e-mail
describing exactly what your workflow is, step-by-step, that might help you get 
better answers
to your questions.

Ok, that's a good idea. Hopefully people haven't [already] muddled the two 
topics of bsdconfig and the separate topic of my workflow @ $work (which 
requires me to understand the nitty gritty details of +MANIFEST).

As you say, I should really spin the latter off into a thread of its own (I 
gave the disclaimer at the beginning that the former was a side-"thing", but it 
seems to have taken over this thread which was supposed to be "hey, let's go 
pkg on bsdconfig").

This thread should stay specific to "pkg_install is dead, long live pkg -- 
dteske needs help answering some questions about pkg for bsdconfig integration 
with it".

Questions still "out-there" in the thread in that arena are:

What are the mirrors (actual addresses) -- one post in this thread asks "hey, 
is this the right _first-step_" in which I proceed to shift away from FTP and 
toward HTTP (removing all the swedish, japanese, etc. FTP mirrors and adding 
at-a-minimum an IPv4 and IPv6 master HTTP mirror).

What's the most efficient way (in a UI) to display a list of packages (we've 
covered rquery; we've covered how rquery works; but I'm thinking to myself... 
if an "rquery" requires an "update", could I perhaps have the UI somehow 
present something better than a "waiting for pkg update" box -- e.g., if I go 
fetch the DB and put it where pkg expects it, I could give progress bars, 
transfer speeds, completion time, etc. etc. in dialog style).

   The FreeBSD ports team has been migrating to pkgng
for the past few years, so a lot of people understand the FreeBSD 
ports/packages workflow.
because it has been discussed for a few years on multiple mailing lists, and
at various BSD conferences like BSDCan.

Indeed. However, I'm looking to create the UI for pkgng, and we're bound to run 
into areas of discussion that haven't been discussed.

I know that some have worked on UIs for pkgng, but those UIs have enjoyed a 
level of integration that I cannot. All the suggestions to use libpkg are not 
that helpful because the code is not in C.

Some of the confusion with others on this list is that you are working on
a tool called "bsdconfig" and committing it into FreeBSD-CURRENT,
but the workflow you are trying to support is slightly different than what
the FreeBSD ports/packages teams have been supporting and adapting
their tools and workflow towards.

The topic of my own personal quest to understand +MANIFEST and the tarball 
contents is entirely separate from the bsdconfig topic. How we ended up down 
this road is by-way of ignoring the sign posted, which said (as a pre-qualifier 
to the text that introduced this subject) "This is going to be important not 
for bsdconfig, but $work" (highlight "not for bsdconfig").

Here is the original quote (which garnered flame, but folks like Matthew Seaman 
took me seriously and -- through it should have been pushed to a new thread -- 
was answering this little foot-note which I had at the bottom of my July 13th, 
10:48P PDT response in this thread):

(begin quote)
On Jul 13, 2013, at 10:48 PM, Teske, Devin wrote:
Question: Where can I learn more about the actual format of what's in the new 
tarballs? This is going to be important not for bsdconfig, but $work (we have 
our own build platform; I'm going to have to rewrite it from mastering PLIST 
files to mastering YAML MANIFEST files and I want to know all the gritty 
(end quote)

So just to clear things up... [again]

bsdconfig needs to blaze the trail on building a UI around pkg.

My own personal quest to understand +MANIFEST so I can update my "pkgbase" 
(which is like ports)...
... not related to bsdconfig.

I think that if you better describe your workflow in a separate e-mail,
then folks can better recommend how to adapt bsdconfig
to the new pkg tools.


  You may need to be open to using
things like libpkg inside bsdconfig, if that is the best solution for bsdconfig 
inside FreeBSD.

Not C.

I agree with you that having a pkgng transition document on the wiki
would be useful for systems integrators.

That was actually one of my original points is all. And the very first reply 
was about ... "8/9 packages?"

My response to "pkg_install is dead, long live pkg" (paraphrasing) was actually 
just about wanting a transition document (of which I would then digest in my 
integration of bsdconfig). However, I was more than happy to turn the thread 
into a "let's get bsdconfig updated" thread.

  I know that many
people build products based on FreeBSD and leverage the pkg_XXX tools
for building their own products.  I have done this myself in the past.
Clarifying the transition path for system integrators would be very helpful.

I'm still trying to figure out how our eccentricities here @ work can translate 
back. pkgbase solves our problems (in a nut-shell) because we're slow and we 
need something that allows us to dance around problems without worrying about 
what the out-side world is doing. When we come up for air, we rally around the 
latest software, and then go back under water and don't see the light of day 
until we've delivered a solid product.

It almost seems like *everybody* else wants to do the opposite. Which is 
whenever they hit a problem, go immediately to the ports three, update, and see 
if the problem goes away. Unfortunately, with over a million lines of 
source-code in our product and touching everything from kernel source to 
user-land _and_ running on 3 classes of hardware (server, workstation, and 
pedestal), __AND__ interfacing with USB devices, high-end SCSI scanners, and 
many custom pieces of hardware...

... when we do come up for air in-between major releases, we have zero time to 
waste in regression testing.

Maybe you can help write the transition doc, since you have the perspective?

I think that when bsdconfig has been updated to work with pkg... that would be 
an *excellent* time for me to jump in on a doc-writing event. It will be fresh 
in my mind *and* we can go through subversion revision history to see exactly 
what was changed (for example, the communicated list of mirrors would be worth 
documenting in a Wiki alone, imho).

The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.
freebsd-current@freebsd.org mailing list
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Reply via email to