Re: `make cleanworld` and /usr/obj/lib32/usr/src
On Sat, 27 Feb 2010 14:49:58 +0100 (CET) Alexander Best alexbes...@wwu.de wrote: how about something like this? Based on a quick look it seems OK to me. Have you tested it? --- Gary Jennejohn ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: `make cleanworld` and /usr/obj/lib32/usr/src
Gary Jennejohn schrieb am 2010-02-28:
On Sat, 27 Feb 2010 14:49:58 +0100 (CET)
Alexander Best alexbes...@wwu.de wrote:
how about something like this?
Based on a quick look it seems OK to me.
Have you tested it?
i did. it works when `make cleanworld` is being run from /usr/src. i'm not
sure however what the
.if ${.CURDIR} == ${.OBJDIR} || ${.CURDIR}/obj == ${.OBJDIR}
.if exists(${BW_CANONICALOBJDIR}/)
statement is for so i didn't include the lib32 cleanup in that clause. my
guess is that it's used to catch the case where `make cleanworld` is being run
in /usr/obj/usr/src
cheers.
alex
---
Gary Jennejohn
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [Proof of Concept] Stacked unionfs based 'tinderbox'
Hi,
Apologies for the delayed reply.
On Thursday 25 February 2010 15:53:03 Ulrich Spörlein wrote:
On Thu, 25.02.2010 at 10:08:15 +0200, David Naylor wrote:
Hi,
As some may have noticed on -current I have been working on using
stacked unionfs to implement a 'tinderbox' type build system. I have
successfully used the scripts to build x11/xorg (and have compared the
results to using the traditional approach using pkg_add). The build
system is stable except for one nasty corner case: deadlocks.
When I did this a couple of years ago, the major problems were failing
chdir(2) calls during ports build, etc.
I'm hoping time and -current will solve this problem as it is beyond my
ability to fix.
To setup a compatible test environment requires:
- recompile the kernel with `options KSTACK_PAGES=32`, otherwise the
kernel will panic with a double fault. WITNESS options results in
substantial performance degradation.
- patch mtree (see below) [optional]
- create the appropriate chroot environment (and reboot) [see below
for corner case]
A performance bottleneck in mtree was identified. This resulted in
mtree (as run by port install) consuming ~20% of the build time. See
bin/143732 for a patch and further details.
Good work!
The normal tinderbox approach takes ~80% more time to install compared to
the quick and dirty approach. The stacked unionfs approach takes ~170%
more time (an increase of ~50% over the tinderbox approach). Some
performance gains can be had if one uses memory backed storage (vs HDD
in this case).
Please explain: what is the quick and dirty approach and which one is
faster now?
The quick and dirty is `make -C /usr/ports/x11/xorg install clean`. The
stacked unionfs is still the slowest (even with a 20% improvement from
patching mtree).
If one is interested in performance in building ports in a sandbox then the
tinderbox route is the way to go.
As your scripts did not make it through, perhaps you can upload them to
the wiki? What I did back then was using a clean base system as the
underlying unionfs store to avoid re-generating the clean base over and
over again. Nowadays, a ZFS clone would probably be the way to go.
I've attached the scripts with a .txt suffix. This should hopefully make it
past mailman. To which wiki page do you refer? You are welcome to add the
scripts there.
I skipped the base system and only unionfs /usr/local. I never cleaned the
base system. I figured that was the easiest to implement when the script runs
in a chroot. An area for further improvements?
I'm not sure if a recursive approach is feasible here, as you can have
only one underlying unionfs mount. But special casing, e.g., perl may
still give a massive speedup. So for each port that has perl as
dependancy, you would not pull in the clean base + pkg_add perl, but
instead grab the clean-base+perl directory as an underlying unionfs.
Have you done any speed comparisons of running programs (such as perl) from a
unionfs. If the disk cache is big enough I don't think there will be a big
performance hit? It is certainly a viable approach.
Regards,
David
#!/bin/sh
BUILDDIR=${BUILDDIR:-/usr/build}
LOCALBASE=${LOCALBASE:-/usr/local}
PORTSDIR=${PORTSDIR:-/usr/ports}
PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports}
PKG_DBDIR=${PKG_DBDIR:-$LOCALBASE/db_pkg}
PACKAGES=${PACKAGES:-$BUILDDIR/packages}
ENV=env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR
PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES
MAKE=$ENV make
PKG_ADD=$ENV pkg_add
PKG_DELETE=$ENV pkg_delete
set -e
mkdir -p $BUILDDIR $PACKAGES
port2name() {
echo $1 | sed 's|[/.-]|_|g'
}
port2pkg() {
local pkg_name=
local port=
port=$1; shift
eval pkg_name=PKG$(port2name $port)
eval pkg=\$$pkg_name
if [ -z $pkg ]
then
pkg=$($MAKE -C $port -V PKGNAME)
eval $pkg_name=$pkg
fi
}
depends() {
local depend=
local depends_name=
local _deps=
local name=
local port=
local type
type=$1
port=$2
eval depends_name=DEPEND_${type}_$(port2name $port)
eval deps=\\$$depends_name\
if [ -z $deps ]
then
echo Getting $type dependancies for $port /dev/stderr
if [ $type = build ]
then
depend_list=$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V
LIB_DEPENDS -V RUN_DEPENDS)
else
depend_list=$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)
fi
for depend in $depend_list
do
name=$(echo $depend | cut -f 2 -d ':')
depends runtime $name
_deps=$_deps $deps $name
done
deps=
for depend in $_deps
do
if [ -z `echo $deps | grep $depend ` ]
then
deps=$deps$depend
fi
done
[ `echo $deps | tr ' ' '\n' | sort` = `echo $deps | tr ' ' '\n' | sort
-u` ]
depends_name=$depends_name
eval $depends_name=\$deps \
fi
}
run() {
set +e
trap echo Terminating... INT TERM EXIT
$@
_status=$?
[ $status
namei() returns EISDIR for / (Re: svn commit: r203990 - head/lib/libc/sys)
[Moving discussion to -hackers. This is not directly related to the
original commit anymore.]
On 2010-02-26, Bruce Evans wrote:
http://people.freebsd.org/~jh/patches/lookup-root.diff
This is in relookup(). I think relookup() is only called from rename(),
so the failing case is unreachable
I don't think it's true for unionfs. Looking at the code it seems that
it can call relookup() also with CREATE and DELETE nameiops.
% @@ -3618,9 +3618,6 @@ kern_renameat(struct thread *td, int old
% if (fromnd.ni_vp-v_type == VDIR)
% tond.ni_cnd.cn_flags |= WILLBEDIR;
% if ((error = namei(tond)) != 0) {
% - /* Translate error code for rename(dir1, dir2/.). */
% - if (error == EISDIR fvp-v_type == VDIR)
% - error = EINVAL;
I think this has nothing to do with the root directory (as its comment says),
but it is to translate the EISDIR returned by ufs_lookup(), etc., when `tond'
is for a (directory) pathname ending in .. So it should not be removed,
except possibly after changing ufs_lookup(), etc., to return EINVAL. The
EISDIR in ufs_lookup() is only for RENAME, so it is strange that any
translation is needed. I apparently put the translation here to avoid
looking at all leaf file systems.
After r199137 the case of dir2/. is handled in lookup() before the
VOP_LOOKUP() call. I am not sure if it should be removed but it seems
that there's no need for the translation after r199137.
--
Jaakko
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: namei() returns EISDIR for / (Re: svn commit: r203990 - head/lib/libc/sys)
Bruce Evans said that the doesn't like this comment you added in your patchset /* * Check for which is a way of talking about the root directory. * We can't provide a parent node for CREATE, DELETE and RENAME * operations. */ and would like to keep / the way it is instead of stripping the slash. however this comment /* * Replace multiple slashes by a single slash and trailing slashes * by a null. This must be done before VOP_LOOKUP() because some * fs's don't know about trailing slashes. Remember if there were * trailing slashes to handle symlinks, existing non-directories * and non-existing files that won't be directories specially later. */ says that some fs's can't handle trailing slashes. how exactly can the root dir be expressed on these fs's? is it in fact ? and which fs's exactly are lacking support for trailing slashes? cheers. alex ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: namei() returns EISDIR for / (Re: svn commit: r203990 - head/lib/libc/sys)
i have a small test app to check {rm|mk}dir()'s errnos with certain args like
/, ., /proc and non-empty dirs. i'll submit it to this thread as soon as i
also add testcases for syscalls like rename(), unlink(), etc.
most of the errno codes returned after applying your patch look correct. i
wonder however why rmdir(/proc) returns EACCESS as unprivileged user.
wouldn't it make more sense to also return EBUSY? why complain about
permission related matters when even root won't be able to perform the
operation.
also: since namei() takes care of handling the / and . cases couldn't
those checks in kern_rmdirat() and kern_unlinkat() be removed?
it might also be interesting to look at the changes netbsd, openbsd and
dragonflybsd have made to vfs_lookup.c. just had a quick look at revision
1.121 from netbsd and they have split lookup() into loads of smaller functions
(see revision 1.117 and 1.118). seems they have been doing a lot of work here
(using heavy XXX-commenting however).
openbsd hasn't made to many changes to vfs_lookup.c.
matthew dillon seems to have done an incredible job on dragonfly in connection
with vfs_lookup.c. basically they completely got rid of namei() and are now
using nlookup() in commit ad57d0edbfceb0cebfb1dce61490df78fcc4a97. the commit
message is quite long and claims due to this change all syscalls which used to
call namei() have become a lot less complex after switching to nlookup().
right now vfs_lookup.c in dragonfly contains only some legacy code used for
compatibility.
again: an incredible job!
+1 for adapting those changes. ;)
cheers.
alex
btw.: there're a few NAMEI_DIAGNOSTIC ifdefs in vfs_lookup.c to increase
verbosity.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: `make cleanworld` and /usr/obj/lib32/usr/src
On Sun, 28 Feb 2010 11:54, alexbestms@ wrote:
Gary Jennejohn schrieb am 2010-02-28:
On Sat, 27 Feb 2010 14:49:58 +0100 (CET)
Alexander Best alexbes...@wwu.de wrote:
how about something like this?
Based on a quick look it seems OK to me.
Have you tested it?
i did. it works when `make cleanworld` is being run from /usr/src. i'm not
sure however what the
.if ${.CURDIR} == ${.OBJDIR} || ${.CURDIR}/obj == ${.OBJDIR}
.if exists(${BW_CANONICALOBJDIR}/)
statement is for so i didn't include the lib32 cleanup in that clause. my
guess is that it's used to catch the case where `make cleanworld` is being run
in /usr/obj/usr/src
You can't run make from a obj directory there is no Makefile, Targets or
spoons. ;)
This line here:
BW_CANONICALOBJDIR:=${MAKEOBJDIRPREFIX}${.CURDIR}
Allows the end-user to set MAKEOBJDIRPREFIX in their environment and have
the obj directory somewhere other than CANONICALOBJDIR.
BW is an abbreviation for buildworld.
cheers.
alex
---
Gary Jennejohn
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
--
jhell
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [patch] extending {amd64|i386} cpu info
On Fri, 26 Feb 2010 22:19, alexbestms@ wrote: hi there, i always forget how to decipher model and family from the cpu id. of course there're 3rd party tools to do that for you, but instead i decided to hack the kernel. ;) maybe somebody finds these changes useful. cheers. alex What about the other arch's ? I only see i386 and amd64. I would think if this functionality is going to make it in that it should be done consistantly across the board, the way it is now. -- jhell ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: namei() returns EISDIR for / (Re: svn commit: r203990 - head/lib/libc/sys)
On Sun, Feb 28, 2010 at 5:11 PM, Alexander Best alexbes...@wwu.de wrote:
i have a small test app to check {rm|mk}dir()'s errnos with certain args like
/, ., /proc and non-empty dirs. i'll submit it to this thread as soon as i
also add testcases for syscalls like rename(), unlink(), etc.
most of the errno codes returned after applying your patch look correct. i
wonder however why rmdir(/proc) returns EACCESS as unprivileged user.
wouldn't it make more sense to also return EBUSY? why complain about
permission related matters when even root won't be able to perform the
operation.
Hmm.. good question. POSIX doesn't fully expound on this case
(http://www.opengroup.org/onlinepubs/009695399/functions/rmdir.html),
and either seem possible...
also: since namei() takes care of handling the / and . cases couldn't
those checks in kern_rmdirat() and kern_unlinkat() be removed?
it might also be interesting to look at the changes netbsd, openbsd and
dragonflybsd have made to vfs_lookup.c. just had a quick look at revision
1.121 from netbsd and they have split lookup() into loads of smaller functions
(see revision 1.117 and 1.118). seems they have been doing a lot of work here
(using heavy XXX-commenting however).
openbsd hasn't made to many changes to vfs_lookup.c.
matthew dillon seems to have done an incredible job on dragonfly in connection
with vfs_lookup.c. basically they completely got rid of namei() and are now
using nlookup() in commit ad57d0edbfceb0cebfb1dce61490df78fcc4a97. the commit
message is quite long and claims due to this change all syscalls which used to
call namei() have become a lot less complex after switching to nlookup().
right now vfs_lookup.c in dragonfly contains only some legacy code used for
compatibility.
again: an incredible job!
+1 for adapting those changes. ;)
cheers.
alex
btw.: there're a few NAMEI_DIAGNOSTIC ifdefs in vfs_lookup.c to increase
verbosity.
Thanks,
-Garrett
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [patch] extending {amd64|i386} cpu info
On Sun, Feb 28, 2010 at 5:43 PM, jhell jh...@dataix.net wrote: On Fri, 26 Feb 2010 22:19, alexbestms@ wrote: hi there, i always forget how to decipher model and family from the cpu id. of course there're 3rd party tools to do that for you, but instead i decided to hack the kernel. ;) maybe somebody finds these changes useful. cheers. alex What about the other arch's ? I only see i386 and amd64. I would think if this functionality is going to make it in that it should be done consistantly across the board, the way it is now. This is trivial enough of a change that I think it should be done for amd64/i386. I say that because it may be a bit more difficult for arm, mips, powerpc, sparc64, etc (probably not as much...). Thanks, -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
