Re: poudriere builds all dependencies, even if packages have already been built
On 4/3/2013 11:17, Stefan Bethke wrote: Firstly, is there a better mailing list to discuss poudriere? I'm just getting my feet wet with poudriere, and are a bit surprised that all dependencies for a port are built, instead of installing the packages that got built earlier already. Is this intentional? Is there a configuration option to have the requisite packages installed before building a port? That's not normal behavior. When I saw something like that, it turned out to be a bug in the bsd.port.mk but that should not apply for you. There's a #poudriere channel at freenode IRC. John ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere builds all dependencies, even if packages have already been built
On Wed, Apr 03, 2013 at 11:20:15AM +0200, John Marino wrote: On 4/3/2013 11:17, Stefan Bethke wrote: Firstly, is there a better mailing list to discuss poudriere? I'm just getting my feet wet with poudriere, and are a bit surprised that all dependencies for a port are built, instead of installing the packages that got built earlier already. Is this intentional? Is there a configuration option to have the requisite packages installed before building a port? That's not normal behavior. When I saw something like that, it turned out to be a bug in the bsd.port.mk but that should not apply for you. There's a #poudriere channel at freenode IRC. John ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org This mailing list is fine also p...@freebsd.org is more accurate for this. And no it is not normal poudriere should always reuse previously built packages if they are built for the same combinaison jailname/portstreename/setname. regards, Bapt pgpjKtGMa0mUW.pgp Description: PGP signature
Re: poudriere builds all dependencies, even if packages have already been built
On 4/3/2013 4:17 AM, Stefan Bethke wrote: Firstly, is there a better mailing list to discuss poudriere? I'm just getting my feet wet with poudriere, and are a bit surprised that all dependencies for a port are built, instead of installing the packages that got built earlier already. Is this intentional? Is there a configuration option to have the requisite packages installed before building a port? Thanks, Stefan Here or pkg@ is fine. This is exactly what it does. It builds the dependencies one time then installs then builds the port needing them. On the next run if the dependencies have not updated it reuses them. However, if a dependency is updated then everything depending on it will be rebuilt as well to be more safe in case a PORTREVISION bump was missed in cases of shlib bumps or similar. -- Regards, Bryan Drewery bdrewery@freenode/EFNet signature.asc Description: OpenPGP digital signature
Re: poudriere builds all dependencies, even if packages have already been built
Am 03.04.2013 um 12:53 schrieb Bryan Drewery:
On 4/3/2013 4:17 AM, Stefan Bethke wrote:
Firstly, is there a better mailing list to discuss poudriere?
I'm just getting my feet wet with poudriere, and are a bit surprised that
all dependencies for a port are built, instead of installing the packages
that got built earlier already. Is this intentional? Is there a
configuration option to have the requisite packages installed before
building a port?
Thanks,
Stefan
Here or pkg@ is fine.
This is exactly what it does. It builds the dependencies one time then
installs then builds the port needing them. On the next run if the
dependencies have not updated it reuses them.
That's what I would expect, but that's not what I'm seeing.
I've been running build incrementally, fixing a problem, then running bulk
again. As expected, only the packages that hadn't been built successfully yet
were built. However, dependencies for these ports are not installed from the
already created packages, but are recompiled again. In this build log, you can
see that pkg is installed from the port instead of from the package; I seem to
get this for a large number of builds. I wouldn't really mind except for the
enormous amount of time it takes to rebuild the same ports over and over again.
I've had some trouble with poudriere not mounting things where they belong. I
manually created /usr/ports and /wrkdirs and recreated the clean snapshot.
This worked around the mount: no such file or directory problems, but I guess
there might be an underlying cause to these and a couple more issues. How
would I go about identifying these? I think I followed the installation
instructions very carefully.
$ head -100 ImageMagick-6.8.0.7_1.log
build started at Wed Apr 3 07:31:08 UTC 2013
port directory: /usr/ports/graphics/ImageMagick
building for: 9.1-STABLE amd64
maintained by: c.petrik.s...@gmail.com
Makefile ident: $FreeBSD: head/graphics/ImageMagick/Makefile 315309
2013-03-26 15:55:59Z bapt $
Poudriere version: 2.4.1
---Begin Environment---
BLOCKSIZE=K
MAIL=/var/mail/root
STATUS=1
PKG_EXT=txz
FORCE_PACKAGE=yes
tpid=97139
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin
POUDRIERE_BUILD_TYPE=bulk
NBPARALLEL=4
PKGNG=1
PKG_DELETE=/usr/local/sbin/pkg delete -y -f
PKG_ADD=/usr/local/sbin/pkg add
PWD=/root/eisenboot
LOGS=/var/poudriere/data/logs
HOME=/root
USER=root
SKIPSANITY=0
LOCALBASE=/usr/local
PACKAGE_BUILDING=yes
VERSION=2.4.1
---End Environment---
---Begin make.conf---
/root/eisenboot/poudriere.d/make.conf
WITH_PKGNG=yes
# Das sollte eigentlich poudriere selber machen
WRKDIRPREFIX?= /wrkdirs
WRKDIR?=${WRKDIRPREFIX}${.CURDIR:S/${PORTSDIR}//}
MASTER_SITE_BACKUP?=\
http://mirror.hanse.de/pub/FreeBSD/ports/distfiles/${DIST_SUBDIR}/
MASTER_SITE_OVERRIDE?= ${MASTER_SITE_BACKUP}
# Port-Konfiguration
openvpn_UNSET= EASYRSA
PACKAGE_BUILDING=yes
DISABLE_MAKE_JOBS=yes
---End make.conf---
---Begin OPTIONS List---
=== The following configuration options are available for
ImageMagick-6.8.0.7_1:
16BIT_PIXEL=on: 16bit pixel support
BZIP2=on: bzip2 compression support
DJVU=off: DJVU format support (needs THREADS)
DOCS=on: Build and/or install documentation
FFTW=on: Discrete Fourier Transform support
FONTCONFIG=on: X11 font configuration support
FPX=on: FlashPix image format support
FREETYPE=on: TrueType font rendering support
GRAPHVIZ=off: Graphviz support
GSLIB=off: libgs (Postscript SHLIB) support
HDRI=off: High dynamic range images support
JASPER=on: JPEG 2000 support via JasPer
JBIG=on: JBIG image format support
JPEG=on: JPEG image format support
LCMS2=on: LCMS2 support
LQR=on: Liquid Rescale support
LZMA=on: LZMA compression support
MODULES=on: Modules support
OPENEXR=off: HDR image format support via OpenEXR
OPENMP=off: Parallel processing support via OpenMP
PANGO=off: Pango rendering library support
PDF=on: PDF document support
PERL=on: Perl scripting language support
PNG=on: PNG image format support
SVG=on: SVG vector image format support
TESTS=off: Run bundled self-tests after build
THREADS=off: Threading support
TIFF=on: TIFF image format support
WEBP=on: WebP image format support
WMF=on: Windows Metafile support
=== Use 'make config' to modify these settings
---End OPTIONS List---
===phase: depends ==
=== ImageMagick-6.8.0.7_1 depends on file: /usr/local/sbin/pkg - not found
===Verifying install for /usr/local/sbin/pkg in /usr/ports/ports-mgmt/pkg
=== License BSD accepted by the user
=== Fetching all distfiles required by pkg-1.0.11 for building
=== Extracting for pkg-1.0.11
=== License BSD accepted by the user
=== Fetching all distfiles required by pkg-1.0.11 for building
= SHA256 Checksum OK for
Re: poudriere builds all dependencies, even if packages have already been built
On 4/3/2013 6:26 AM, Stefan Bethke wrote:
Am 03.04.2013 um 12:53 schrieb Bryan Drewery:
On 4/3/2013 4:17 AM, Stefan Bethke wrote:
Firstly, is there a better mailing list to discuss poudriere?
I'm just getting my feet wet with poudriere, and are a bit surprised that
all dependencies for a port are built, instead of installing the packages
that got built earlier already. Is this intentional? Is there a
configuration option to have the requisite packages installed before
building a port?
Thanks,
Stefan
Here or pkg@ is fine.
This is exactly what it does. It builds the dependencies one time then
installs then builds the port needing them. On the next run if the
dependencies have not updated it reuses them.
That's what I would expect, but that's not what I'm seeing.
I've been running build incrementally, fixing a problem, then running bulk
again. As expected, only the packages that hadn't been built successfully
yet were built. However, dependencies for these ports are not installed from
the already created packages, but are recompiled again. In this build log,
you can see that pkg is installed from the port instead of from the package;
I seem to get this for a large number of builds. I wouldn't really mind
except for the enormous amount of time it takes to rebuild the same ports
over and over again.
If you're building pkgng packages you'll also need pkgng in the host
installed. (Not converted, just installed). If you're missing pkgng on
the host, install that and build again. The 2nd build should be ok.
I've had some trouble with poudriere not mounting things where they belong.
I manually created /usr/ports and /wrkdirs and recreated the clean
snapshot. This worked around the mount: no such file or directory
problems, but I guess there might be an underlying cause to these and a
couple more issues. How would I go about identifying these? I think I
followed the installation instructions very carefully.
I have not heard of these issues and it appears your setup is now messed
up. None of this manual hacking should be needed. I suggest recreating
your jail.
$ head -100 ImageMagick-6.8.0.7_1.log
build started at Wed Apr 3 07:31:08 UTC 2013
port directory: /usr/ports/graphics/ImageMagick
building for: 9.1-STABLE amd64
maintained by: c.petrik.s...@gmail.com
Makefile ident: $FreeBSD: head/graphics/ImageMagick/Makefile 315309
2013-03-26 15:55:59Z bapt $
Poudriere version: 2.4.1
---Begin Environment---
BLOCKSIZE=K
MAIL=/var/mail/root
STATUS=1
PKG_EXT=txz
FORCE_PACKAGE=yes
tpid=97139
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin
POUDRIERE_BUILD_TYPE=bulk
NBPARALLEL=4
PKGNG=1
PKG_DELETE=/usr/local/sbin/pkg delete -y -f
PKG_ADD=/usr/local/sbin/pkg add
PWD=/root/eisenboot
LOGS=/var/poudriere/data/logs
HOME=/root
USER=root
SKIPSANITY=0
LOCALBASE=/usr/local
PACKAGE_BUILDING=yes
VERSION=2.4.1
---End Environment---
---Begin make.conf---
/root/eisenboot/poudriere.d/make.conf
WITH_PKGNG=yes
# Das sollte eigentlich poudriere selber machen
WRKDIRPREFIX?=/wrkdirs
WRKDIR?= ${WRKDIRPREFIX}${.CURDIR:S/${PORTSDIR}//}
MASTER_SITE_BACKUP?= \
http://mirror.hanse.de/pub/FreeBSD/ports/distfiles/${DIST_SUBDIR}/
MASTER_SITE_OVERRIDE?=${MASTER_SITE_BACKUP}
# Port-Konfiguration
openvpn_UNSET=EASYRSA
PACKAGE_BUILDING=yes
DISABLE_MAKE_JOBS=yes
---End make.conf---
---Begin OPTIONS List---
=== The following configuration options are available for
ImageMagick-6.8.0.7_1:
16BIT_PIXEL=on: 16bit pixel support
BZIP2=on: bzip2 compression support
DJVU=off: DJVU format support (needs THREADS)
DOCS=on: Build and/or install documentation
FFTW=on: Discrete Fourier Transform support
FONTCONFIG=on: X11 font configuration support
FPX=on: FlashPix image format support
FREETYPE=on: TrueType font rendering support
GRAPHVIZ=off: Graphviz support
GSLIB=off: libgs (Postscript SHLIB) support
HDRI=off: High dynamic range images support
JASPER=on: JPEG 2000 support via JasPer
JBIG=on: JBIG image format support
JPEG=on: JPEG image format support
LCMS2=on: LCMS2 support
LQR=on: Liquid Rescale support
LZMA=on: LZMA compression support
MODULES=on: Modules support
OPENEXR=off: HDR image format support via OpenEXR
OPENMP=off: Parallel processing support via OpenMP
PANGO=off: Pango rendering library support
PDF=on: PDF document support
PERL=on: Perl scripting language support
PNG=on: PNG image format support
SVG=on: SVG vector image format support
TESTS=off: Run bundled self-tests after build
THREADS=off: Threading support
TIFF=on: TIFF image format support
WEBP=on: WebP image format support
WMF=on: Windows Metafile support
=== Use 'make config' to modify these settings
Re: poudriere builds all dependencies, even if packages have already been built
Am 03.04.2013 um 13:30 schrieb Bryan Drewery:
I've had some trouble with poudriere not mounting things where they belong.
I manually created /usr/ports and /wrkdirs and recreated the clean
snapshot. This worked around the mount: no such file or directory
problems, but I guess there might be an underlying cause to these and a
couple more issues. How would I go about identifying these? I think I
followed the installation instructions very carefully.
I have not heard of these issues and it appears your setup is now messed
up. None of this manual hacking should be needed. I suggest recreating
your jail.
OK, sound like a plan.
I'm getting further that I used to, but I do get an error when I try to kick
off the bulk build (which poudriere seems to ignore), see below:
Here's my poudriere.conf:
# egrep -v '^(#.*)?$' /usr/local/etc/poudriere.conf
ZPOOL=tank
FREEBSD_HOST=http://ftp2.de.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/var/poudriere
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=/freebsd/distfiles
SVN_HOST=mirror.hanse.de
CCACHE_DIR=${BASEFS}/ccache
NO_RESTRICTED=no
According to
https://fossil.etoilebsd.net/poudriere/doc/trunk/doc/use_system_ports_tree.wiki,
I've marked my ports tree zfs:
# zfs get all tank/freebsd/checkout
NAME PROPERTY VALUE SOURCE
tank/freebsd/checkout type filesystem -
...
tank/freebsd/checkout poudriere:namesystemportslocal
tank/freebsd/checkout poudriere:typeports local
Created a jail for 9.1-RELEASE:
# poudriere jail -c -j 91amd64 -v 9.1-RELEASE
cannot open 'tank/poudriere': dataset does not exist
Creating 91amd64 fs... done
Fetching base.txz for FreeBSD 9.1-RELEASE amd64
/var/poudriere/jails/91amd64/fromftp/base.txz 100% of 57 MB 9122 kBps
Extracting base.txz... done
Fetching src.txz for FreeBSD 9.1-RELEASE amd64
/var/poudriere/jails/91amd64/fromftp/src.txz 100% of 91 MB 10 MBps
Extracting src.txz... done
Fetching games.txz for FreeBSD 9.1-RELEASE amd64
/var/poudriere/jails/91amd64/fromftp/games.txz100% of 1091 kB 3878 kBps
Extracting games.txz... done
Fetching lib32.txz for FreeBSD 9.1-RELEASE amd64
/var/poudriere/jails/91amd64/fromftp/lib32.txz100% of 9515 kB 7865 kBps
Extracting lib32.txz... done
Cleaning up... done
Jail 91amd64 9.1-RELEASE amd64 is ready to be used
# cat /usr/local/etc/poudriere.d/make.conf
# General configuration
WITH_PKGNG=yes
# Port-specific configuration
openvpn_UNSET= EASYRSA
Start building my list of ports:
# poudriere bulk -f poudriere-default-ports -j 91amd64 -J 3 -p systemports
Mounting system devices for 91amd64
/etc/resolv.conf - /var/poudriere/jails/91amd64/etc/resolv.conf
Starting jail 91amd64
realpath: /freebsd/checkout/ports/distfiles: No such file or directory
Mounting ports from: /freebsd/checkout/ports
Mounting packages from: /var/poudriere/data/packages/91amd64-systemports
Appending to /etc/make.conf: /root/eisenboot/poudriere.d/make.conf
Calculating ports order and dependencies
Sanity checking the repository
Deleting stale symlinks
Cleaning the build queue
Hit ctrl+t at any time to see build progress and stats
Building 244 packages using 3 builders
^C
Not sure what the problem is:
# realpath /freebsd/checkout/ports/distfiles
/freebsd/checkout/ports/distfiles
And I do have pkg installed in the host:
# pkg_info pkg-*
Information for pkg-1.0.11:
Comment:
New generation package manager
Description:
New Generation package management tool for FreeBSD
WWW: http://wiki.freebsd.org/pkgng
# which -a pkg
/usr/sbin/pkg
/usr/local/sbin/pkg
I'm letting the build run for a while, and will inspect the logs.
Thanks for your help,
Stefan
--
Stefan Bethke s...@lassitu.de Fon +49 151 14070811
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere builds all dependencies, even if packages have already been built
Am 03.04.2013 um 14:02 schrieb Stefan Bethke: (Setup with system ports tree and FTP-installed 9.1-RELEASE excluded) Starting jail 91amd64 realpath: /freebsd/checkout/ports/distfiles: No such file or directory Mounting ports from: /freebsd/checkout/ports Mounting packages from: /var/poudriere/data/packages/91amd64-systemports Appending to /etc/make.conf: /root/eisenboot/poudriere.d/make.conf Calculating ports order and dependencies Sanity checking the repository Deleting stale symlinks Cleaning the build queue Hit ctrl+t at any time to see build progress and stats Building 244 packages using 3 builders ^C Not sure what the problem is: # realpath /freebsd/checkout/ports/distfiles /freebsd/checkout/ports/distfiles And I do have pkg installed in the host: # pkg_info pkg-* Information for pkg-1.0.11: Comment: New generation package manager Description: New Generation package management tool for FreeBSD WWW: http://wiki.freebsd.org/pkgng # which -a pkg /usr/sbin/pkg /usr/local/sbin/pkg I'm letting the build run for a while, and will inspect the logs. On the second bulk run, the above error did not occur. I successfully built a complete set of packages, then startd over with -c, interrupted the build with ^C and continued it, all successfully. I'll try and figure out what it was specifically that made it break beforehand (I was make world-installing 9-stable, and svn co a fresh ports tree from a local mirror). Stefan -- Stefan Bethke s...@lassitu.de Fon +49 151 14070811 ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: poudriere builds all dependencies, even if packages have already been built
Am 03.04.2013 um 16:29 schrieb Stefan Bethke s...@lassitu.de: Am 03.04.2013 um 14:02 schrieb Stefan Bethke: (Setup with system ports tree and FTP-installed 9.1-RELEASE excluded) Starting jail 91amd64 realpath: /freebsd/checkout/ports/distfiles: No such file or directory Mounting ports from: /freebsd/checkout/ports On the second bulk run, the above error did not occur. I successfully built a complete set of packages, then startd over with -c, interrupted the build with ^C and continued it, all successfully. I'll try and figure out what it was specifically that made it break beforehand (I was make world-installing 9-stable, and svn co a fresh ports tree from a local mirror). I've now retraced my steps from yesterday, but can't recreate the problem, so all seems to be fine. When I created a new jail (svn checkout of stable/9), on the first bulk run I got the realpath error again; it seems poudriere fixes it itself? Stefan -- Stefan Bethke s...@lassitu.de Fon +49 151 14070811 ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
